3 ms·
This stuff has been making the rounds in my social circle who mostly do infra things. DHH is a salesman/marketer above all else so you need to take anything he
by jpgvm 3y ago
This stuff has been making the rounds in my social circle who mostly do infra things.
DHH is a salesman/marketer above all else so you need to take anything he says with a grain of salt.
That said he is right on several crucial advantages of running your own kit:
1. It's a lot cheaper, once you reach the scale of 2-3 racks (or roughly 30-45kw of gear or so) you are well in the green even if you need to hire additional staff to manage the HW side of things.
2. Performance is on a completely different level especially if your workload benefits from I/O. Being able to stack at 2U box with an absolutely silly amount of NVMe drives makes light work of what would otherwise be tricky scaling challenges.
3. Access to real raw networking makes a whole bunch of things more optimal, namely real BGP (yay anycast) but also better load balancing primitives like Maglev.
Now these are all very good reasons to want to be on your own gear but DHH completely glosses over the very real disadvantages.
1. As time goes on you need new hardware, either to replace gear that is out of service life, has otherwise failed or most commonly for expansion. This will require doing the same validation you did when you first ported off (or built out if you were greenfield) because it's very likely you aren't going to be able to get the same SKUs with the same BOM. So over time your infrastructure becomes less and less homogenous which increases the cost of management and becomes a burden in terms of operational knowledge. TLDR introducing new hardware never gets easier, the more unique SKUs you add the less fun things become.
2. Networking is generally entirely glossed over in posts like these. Running your own network with your own AS, IP space, edge routing, dealing with peering and transit, balancing upstreams etc is work. You might be able to offload this to your DC provider in some cases but usually you are going to have real network engineers on staff, you can't repurpose a curious programmer into a network engineer (atleast quickly). That is also ignoring the fact that most network gear still hasn't caught up to the age of IaC. If you want that you will be spending either more money (Arista/friends) or paving road ahead of yourself (unless things have changed in the last few years I have been away from gear).
3. The financial aspect is as a whole a lot cheaper but you trade total net $ for complexity of financial arrangements. Often times you will want to lease servers, you need DC leases, you need transit agreements etc. You went from a single entity you pay in "The Cloud" to many different vendors. Vendors suck of course, lots of vendors sucks even more. If you are a larger company with a procurement office this is a non-issue or can even be an advantage because said procurement team can put the screws into vendors individually.. so YMMV depending on size of your company.
4. While most older software engineers and infra folk generally have bare HW experience the newer generation (last 10 years) for the most part have never touched anything lower level than a VPS. The days where every infra guy knew IOS, megacli, etc like the back of their hands and wrote Perl one-liners instinctively has sadly passed. You will need to seek out actually qualified engineers. This can put off businessy/less-technical types because they don't like the idea of not being able to hire a bunch of juniors that know close to nothing and will accept peanuts for pay. If you want to to do this but you are coming from the technical side (you want them sweet IOPs) this is probably one of the bigger organisational barriers you will probably face.
4. Things fail. Often at very inopportune times. Runbooks, backups, DR, etc these are all much higher stakes on your own gear. Culturally if you want to go this route you also need to be willing to accept a higher level of investment in these and associated activities like running proper drills. You should do this anyway but in the case of "The Cloud" you are getting higher durability etc that do reduce risk vs your own spinning rust/SSDs and operator error.
All that being said I'm still on team "run your own gear after $200k/mo bills start showing up" but I don't like DHH painting such a one-sided picture of what this really entails.
- Nextgrid 3y agoWhat's your opinion on old-school dedicated server hosting providers? They would handle all the hardware side for you, with an experience close to a virtual machine provider. No need to worry about the details of owning & running actual racks in a DC.
- danpalmer 3y agoI worked for 7 years for a company that was mostly bare-metal rented hardware. The service provider handled half of this stuff, but not everything. Off the top of my head... - Networking was still an issue. We didn't get much control over the network we were on, I think we were on a vlan, but we weren't in control of it. We needed quite a robust firewall setup on all our machines. All machines were on the internet, again not something we could control. - Hardware did become varied over time. We had to do things like creating a new database primary and using the old machine that was now too slow for a read replica. Eventually we found that the read replica was slower than just reading from the primary! - One machine kept eating SSDs, they had a life expectancy of about 3 months in that machine. Neither us or the provider ever figured out why. The replacement process was a hassle and required a lot of manual work on both ends (even though it was in a RAID array designed for hot swapability). - Once the provider gave us a machine with a BMC, but didn't tell us about the BMC or give us any credentials for it. The BMC was on the internet with default creds, it got owned, the machine got owned, and we were running a Monero miner for someone for a while. Thankfully no other damage, but that could have been company ending if we hadn't lucked out on a few details. - The DC had bad peering with an ISP (could have been a customer ISP, or maybe our office, can't remember). That caused a bunch of hassle. Again, not our fault, but definitely our problem. Hard to get someone interested in fixing it. - The provider got acquired by another bigger DC provider. All the experienced staff left, and service dropped off a cliff. This got us to finally move to the cloud. - Hardware lead times are rarely less than a few days, often weeks or months. You have to capacity plan far out, and often make trade-offs like taking a spec you don't want (over and more expensive, or under and less useful) to get hardware when you need it. We might have been able to get the provider to do more for us here, but anything they did was operated at the level of email and someone on their end manually configuring things. It was error prone, it was slow, it was unpredictable. None of this is any criticism of the provider, they were good at what they did, but this is just the inevitable outcome of that model as far as I can tell.