6 ms·

What you can do about it is go back to the early 2000s and rack up your own boxes in a colo, with some reasonable unmetered bandwidth allocations, and skip the

by Syonyk 3y ago

What you can do about it is go back to the early 2000s and rack up your own boxes in a colo, with some reasonable unmetered bandwidth allocations, and skip the whole "cloud" nonsense. My personal box is a 6C/12T Xeon from a few years back, 128GB ECC RAM, and about 24TB usable disk plus 2TB NVMe, for running a lot of other VMs that I run actual services in (remote backup, my web presence, communications, game servers, the works). Then a 25Mbit/100Mbit burst connection at a local datacenter for a bit over $100/mo. A 12vCPU/96GB RAM GCE N2 box with similar specs as my box is about $1800/mo. So... given that "a few months of cloud spend" buys my system, I genuinely don't see the case for "cloud" if you're using a lot of data, or a lot of RAM, or a lot of CPU. Yes, it's managed and convenient, but it's horridly expensive for what you get compared to "owning your own hardware." I'd been spending quite a bit more a month on various cloud instances to do the same thing I do with my server.

jauntywundrkind 3y ago
If you want bandwidth, vps are quite cheap. 100Mbps averages to a a little over 32TB/mo. There there's tons of low CPU low ram systems with 32TB/mo transfer for cheap. Now under $10/mo: https://lowendbox.com/blog/hosteons-they-survived-dedipath-and-are-thriving-in-six-locations/ https://lowendbox.com/blog/hosteons-they-survived-dedipath-a... People should rack their own rpi5 or nucs, at least. There's various providers for both. And consider moving up from there. The remaining folks who will accept & rack a mid tower box for under $75 are saints.
- Syonyk 3y ago
  Sure, I'll use a $5/mo DO Droplet (maybe $6/mo now for the new ones?) and their 1TB transfer pool if I just need raw bandwidth, but, tbh, not a lot of stuff is just pure bandwidth needs. And "capable cloud boxes for various purposes" start adding up. If I just wanted to shovel out a static website, sure. But I can do that a variety of other ways too. Once you start running "real" things, CPU and RAM use start climbing - mostly RAM in my experience. I run a Matrix homeserver - Synapse really struggle with <4GB RAM on the system, so it's one of a few things on a VM with 16GB. I've got a Minecraft and Valheim server running for friends and family - 8-12GB is reasonable for both. You certainly can do it in less, and I've done it, but a lot of problems go away when you can just throw RAM at the problem. Also, most of the cheap VPS boxes are badly IOPS limited on disk - unless you start paying a good bit more for faster disk. I split my box between a 2TB NVMe mirror for VM root drives and "high write traffic" stuff (the gameserver drives and the like), and my large, slow, spinning rust array for backup storage and "big stuff." Again, you can do it with cloud, but storage adds up in a hurry - 24TB of "slow" disk is a grand a month. It just depends on what you need. I really like cloud boxes for light use "toy" boxes, and if that's what you want, they're great. I just cannot wrap my head around the merits of spending a lot of money for "serious cloud compute" when I can build and rack a similarly capable box for 2-3 months of the cloud spend on a similar system. I understand opex vs capex issues for business, but, man, it sure seems stupid to me. Then again, I'm the kind of person who's had a box racked up a colo (or somewhere similar - random basements on a good connection are nice) for the majority of my adult life. So I'm pretty clearly biased. But I did do a few years of "Hey, cloud, this is cool" - and eventually sat down, evaluated my needs and the costs, and went back to a colo'd box.
  - jauntywundrkind 3y ago
    The point of this article was trying to liberate bandwidth, and I was highlighting how even common answers like your Droplet are far from the value-end of what's available. I showed an 18x better price point than what you would have gone to, on transfer alone (the 8GB memory is also a colossal advantage vs 0.5/1gb on a droplet). I think for a lot of people, a nuc with dual m.2 drives is probably a much more affordable & realistic entry point that would probably survive even your fairly demanding gameserving use cases. And cost less than half as much. Overall I'm very interested in a future where we do have scale out edge caches, where we do just need modest resources to maintain wide-scale connectivity. Many personal services are not at this scale or not this architecturally complex, but being able to have efficient scalable personal systems would be an interesting ends to move towards. Matrix would be a good example of a service that would be nice to have good fan-out for.
swatcoder 3y ago
Heathen! Not everyone can afford that PhD in enterprise class hardware management you must have earned to pull this off. And besides, what are you going to do when you wake up tomorrow and you're the next FAANG? You'll never scale to 2B DAU on that box and we all need to be ready for that, just in case. I knew a guy who knew a guy who founded a FAANG, so it could happen to you too. You have to be ready! /s
- Syonyk 3y ago
  Oof. Thanks for the /s, I've quite literally heard that argument from people before for "cloud." That you have to design to scale, physical hardware can't scale rapidly, etc. I assume, if I suddenly had reason to scale, I could call my colo and get another slot from them pretty darn quick. They're local and quite responsive, though I do get a feeling their "public shared rack" offering was created to deal with me. I was the first server in it...
- roncesvalles 3y ago
  But seriously, for scaling up, just have a plan for how to move things to the cloud quickly. Containerization and building everything on Kubernetes help.
  - Syonyk 3y ago
    Or don't! Not everything "has to scale." My stuff isn't built to, and I see no reason to do so. It's my personal server. I run some stuff for my convenience, a quiet backwaters community forum, a Matrix chat server, etc. If any of that really starts to ramp up, I can always split it out into its own VM. If it exceeds what my current server can handle (exceedingly unlikely), I'll go rack up another box for it. And if I get a super sudden spike in traffic, something is probably quite weird. The only thing likely to see that is my personal blog site, which I already have cached in Cloudflare with a typical ~60% cache hit rate, going up north of 90% if there's a spike in traffic, because, being a static site, I've told CF to cache "literally everything there." I probably could move most of my stuff to "the cloud" in an evening's sysadmin, but... why? I already went through the process of moving it off the cloud to save myself an awful lot of money.
mknocker 3y ago
I am thinking about doing something similar. Out of curiosity, do you use a reverse proxy web server to protect from malicious attacks ? That`s probably the only part I would think of leaving in the cloud if I were to host my own servers at home.
- Syonyk 3y ago
  What's different about a box hanging out in the cloud with open ports, vs at home or in a colo with open ports? I use CloudFlare to front most of my web stuff for bandwidth diversion - my blog is a purely static Jekyll site, and I've told CF to "cache everything," so if I get hit with a spike of traffic, it isn't hitting my server (much - comments are still locally hosted with a Discourse forum integration, but if that's overloaded it doesn't block page render). My home bandwidth simply isn't up to hosting. But if I can ever get someone to tap the fiber a quarter mile from my place and give me a drop, I'd move to hosting at home.
washadjeffmad 3y ago
> My personal box ... for a bit over $100/mo. Is that just $50/U?
- Syonyk 3y ago
  It's $75 for a 1U slot (which somehow gives me 2U of space) with dual power, and the rest is bandwidth and an IP block (I've got a /29 so I can put different services on different IPs, because that's... just a thing I do - I'm fundamentally a "first decade of the 2000s Linux sysadmin" in how I think).
deleted 3y ago
[deleted]
frankjr 3y ago
> What you can do about it is go back to the early 2000s and rack up your own boxes in a colo, with some reasonable unmetered bandwidth allocations, and skip the whole "cloud" nonsense. Yep, we used to pay about $35/mo for an unmetered 1Gbit 1U slot at our local datacenter. What surprised me (but maybe is a standard these days?) is that you don't even need to install the server physically yourself. You just buy it, ship it to the datacenter, and wait for the remote access.
kccqzy 3y ago
In the early 2000s before Google even IPO'ed, they experimented with such colo with unlimited bandwidth allocations to run their search crawler. It turns out something that's labeled unlimited isn't truly unlimited.
- Syonyk 3y ago
  Man, some of the early "Google before they had their own DC" stories are epic, if you can get an SRE talking at some point.
theLiminator 3y ago
I think that's great for hosting your own stuff. Just personally not something I'd want to do once I really need high availability (power outages/network outages). But I suspect it might actually be a great way for compute bound startups to start for cheap.
- Syonyk 3y ago
  So... Why do you need such high availability that a rare power outage, network outage, or equipment failure is such a big deal? I ask having spent a lot of my college years (and some beyond) trying to run "enterprise grade uptime on a college student budget" - with the appropriate frustration. I've come around to a more casual, "As long as I can reasonably get it back up in a day or two, meh..." way of thinking about various services. It's important to figure out what you need and what you want - a few friends and I once had a stack of rather compute-heavy boxes in a rack, and we ran most of them without a UPS. The provided UPS wasn't enough amps for what we were running, so we just backed up the core stuff and left the crunchers subject to outages. Stuff could restart easily enough without a big deal. It was annoying, but not a big deal. I'd argue that most startups don't need a multi-region, globally backed up infrastructure. And that they'd be a lot better off with less, cashflow-wise. But I guess that's not "Built to scale to 10B users" that VC fund expect to see these days, so... jump through the hoops provided?
  - theLiminator 3y ago
    Customer trust is one of the most important assets, it's hard to earn, and once lost it's incredibly hard to get back. Hosting your own infra makes that risk much higher. Especially if you're providing a critical path service. I'm not saying that you need some overkill multi-region globally backed up setup, it definitely varies and depends on the nature of your business. If what you're doing works great for you, keep doing it!
bombcar 3y ago
The nice key about this is "local datacenter" - it's nice to be able to drive over to your hardware when needed. I've found some nice colo deals but I'm a bit hesitant at being too far away and requiring remote hands, though I should work out exactly how much I'm paying for power ...

by Syonyk 3y ago

jauntywundrkind 3y ago

If you want bandwidth, vps are quite cheap. 100Mbps averages to a a little over 32TB/mo. There there's tons of low CPU low ram systems with 32TB/mo transfer for cheap. Now under $10/mo: https://lowendbox.com/blog/hosteons-they-survived-dedipath-and-are-thriving-in-six-locations/ https://lowendbox.com/blog/hosteons-they-survived-dedipath-a... People should rack their own rpi5 or nucs, at least. There's various providers for both. And consider moving up from there. The remaining folks who will accept & rack a mid tower box for under $75 are saints.

Syonyk 3y ago
Sure, I'll use a $5/mo DO Droplet (maybe $6/mo now for the new ones?) and their 1TB transfer pool if I just need raw bandwidth, but, tbh, not a lot of stuff is just pure bandwidth needs. And "capable cloud boxes for various purposes" start adding up. If I just wanted to shovel out a static website, sure. But I can do that a variety of other ways too. Once you start running "real" things, CPU and RAM use start climbing - mostly RAM in my experience. I run a Matrix homeserver - Synapse really struggle with <4GB RAM on the system, so it's one of a few things on a VM with 16GB. I've got a Minecraft and Valheim server running for friends and family - 8-12GB is reasonable for both. You certainly can do it in less, and I've done it, but a lot of problems go away when you can just throw RAM at the problem. Also, most of the cheap VPS boxes are badly IOPS limited on disk - unless you start paying a good bit more for faster disk. I split my box between a 2TB NVMe mirror for VM root drives and "high write traffic" stuff (the gameserver drives and the like), and my large, slow, spinning rust array for backup storage and "big stuff." Again, you can do it with cloud, but storage adds up in a hurry - 24TB of "slow" disk is a grand a month. It just depends on what you need. I really like cloud boxes for light use "toy" boxes, and if that's what you want, they're great. I just cannot wrap my head around the merits of spending a lot of money for "serious cloud compute" when I can build and rack a similarly capable box for 2-3 months of the cloud spend on a similar system. I understand opex vs capex issues for business, but, man, it sure seems stupid to me. Then again, I'm the kind of person who's had a box racked up a colo (or somewhere similar - random basements on a good connection are nice) for the majority of my adult life. So I'm pretty clearly biased. But I did do a few years of "Hey, cloud, this is cool" - and eventually sat down, evaluated my needs and the costs, and went back to a colo'd box.
- jauntywundrkind 3y ago
  The point of this article was trying to liberate bandwidth, and I was highlighting how even common answers like your Droplet are far from the value-end of what's available. I showed an 18x better price point than what you would have gone to, on transfer alone (the 8GB memory is also a colossal advantage vs 0.5/1gb on a droplet). I think for a lot of people, a nuc with dual m.2 drives is probably a much more affordable & realistic entry point that would probably survive even your fairly demanding gameserving use cases. And cost less than half as much. Overall I'm very interested in a future where we do have scale out edge caches, where we do just need modest resources to maintain wide-scale connectivity. Many personal services are not at this scale or not this architecturally complex, but being able to have efficient scalable personal systems would be an interesting ends to move towards. Matrix would be a good example of a service that would be nice to have good fan-out for.

swatcoder 3y ago

Heathen! Not everyone can afford that PhD in enterprise class hardware management you must have earned to pull this off. And besides, what are you going to do when you wake up tomorrow and you're the next FAANG? You'll never scale to 2B DAU on that box and we all need to be ready for that, just in case. I knew a guy who knew a guy who founded a FAANG, so it could happen to you too. You have to be ready! /s

Syonyk 3y ago
Oof. Thanks for the /s, I've quite literally heard that argument from people before for "cloud." That you have to design to scale, physical hardware can't scale rapidly, etc. I assume, if I suddenly had reason to scale, I could call my colo and get another slot from them pretty darn quick. They're local and quite responsive, though I do get a feeling their "public shared rack" offering was created to deal with me. I was the first server in it...
roncesvalles 3y ago
But seriously, for scaling up, just have a plan for how to move things to the cloud quickly. Containerization and building everything on Kubernetes help.
- Syonyk 3y ago
  Or don't! Not everything "has to scale." My stuff isn't built to, and I see no reason to do so. It's my personal server. I run some stuff for my convenience, a quiet backwaters community forum, a Matrix chat server, etc. If any of that really starts to ramp up, I can always split it out into its own VM. If it exceeds what my current server can handle (exceedingly unlikely), I'll go rack up another box for it. And if I get a super sudden spike in traffic, something is probably quite weird. The only thing likely to see that is my personal blog site, which I already have cached in Cloudflare with a typical ~60% cache hit rate, going up north of 90% if there's a spike in traffic, because, being a static site, I've told CF to cache "literally everything there." I probably could move most of my stuff to "the cloud" in an evening's sysadmin, but... why? I already went through the process of moving it off the cloud to save myself an awful lot of money.

mknocker 3y ago

I am thinking about doing something similar. Out of curiosity, do you use a reverse proxy web server to protect from malicious attacks ? That`s probably the only part I would think of leaving in the cloud if I were to host my own servers at home.

Syonyk 3y ago
What's different about a box hanging out in the cloud with open ports, vs at home or in a colo with open ports? I use CloudFlare to front most of my web stuff for bandwidth diversion - my blog is a purely static Jekyll site, and I've told CF to "cache everything," so if I get hit with a spike of traffic, it isn't hitting my server (much - comments are still locally hosted with a Discourse forum integration, but if that's overloaded it doesn't block page render). My home bandwidth simply isn't up to hosting. But if I can ever get someone to tap the fiber a quarter mile from my place and give me a drop, I'd move to hosting at home.

washadjeffmad 3y ago

> My personal box ... for a bit over $100/mo. Is that just $50/U?

Syonyk 3y ago
It's $75 for a 1U slot (which somehow gives me 2U of space) with dual power, and the rest is bandwidth and an IP block (I've got a /29 so I can put different services on different IPs, because that's... just a thing I do - I'm fundamentally a "first decade of the 2000s Linux sysadmin" in how I think).

deleted 3y ago

[deleted]

frankjr 3y ago

> What you can do about it is go back to the early 2000s and rack up your own boxes in a colo, with some reasonable unmetered bandwidth allocations, and skip the whole "cloud" nonsense. Yep, we used to pay about $35/mo for an unmetered 1Gbit 1U slot at our local datacenter. What surprised me (but maybe is a standard these days?) is that you don't even need to install the server physically yourself. You just buy it, ship it to the datacenter, and wait for the remote access.

kccqzy 3y ago

In the early 2000s before Google even IPO'ed, they experimented with such colo with unlimited bandwidth allocations to run their search crawler. It turns out something that's labeled unlimited isn't truly unlimited.

Syonyk 3y ago
Man, some of the early "Google before they had their own DC" stories are epic, if you can get an SRE talking at some point.

theLiminator 3y ago

I think that's great for hosting your own stuff. Just personally not something I'd want to do once I really need high availability (power outages/network outages). But I suspect it might actually be a great way for compute bound startups to start for cheap.

Syonyk 3y ago
So... Why do you need such high availability that a rare power outage, network outage, or equipment failure is such a big deal? I ask having spent a lot of my college years (and some beyond) trying to run "enterprise grade uptime on a college student budget" - with the appropriate frustration. I've come around to a more casual, "As long as I can reasonably get it back up in a day or two, meh..." way of thinking about various services. It's important to figure out what you need and what you want - a few friends and I once had a stack of rather compute-heavy boxes in a rack, and we ran most of them without a UPS. The provided UPS wasn't enough amps for what we were running, so we just backed up the core stuff and left the crunchers subject to outages. Stuff could restart easily enough without a big deal. It was annoying, but not a big deal. I'd argue that most startups don't need a multi-region, globally backed up infrastructure. And that they'd be a lot better off with less, cashflow-wise. But I guess that's not "Built to scale to 10B users" that VC fund expect to see these days, so... jump through the hoops provided?
- theLiminator 3y ago
  Customer trust is one of the most important assets, it's hard to earn, and once lost it's incredibly hard to get back. Hosting your own infra makes that risk much higher. Especially if you're providing a critical path service. I'm not saying that you need some overkill multi-region globally backed up setup, it definitely varies and depends on the nature of your business. If what you're doing works great for you, keep doing it!

bombcar 3y ago

The nice key about this is "local datacenter" - it's nice to be able to drive over to your hardware when needed. I've found some nice colo deals but I'm a bit hesitant at being too far away and requiring remote hands, though I should work out exactly how much I'm paying for power ...