5 ms·
I don't know how every company does it. But I led the datacenter and networking teams at Square from 2011 to 2017. In that time we went from 1 DC cage with 4 se
by kelp 5y ago
I don't know how every company does it. But I led the datacenter and networking teams at Square from 2011 to 2017. In that time we went from 1 DC cage with 4 server racks to 4 US DCs, one in a Japan and a couple of network pops. A bit more than 100 server racks total.
My team had 2 guys doing SiteOps. They would travel to the various DCs in the Bay Area and Virginia and do all the maintenance, new installs, etc. And sometimes we'd lean on the colocation remote hands to do a few things.
We had about 5 network engineers, that also handled the corporate network. (12 offices and a network backbone that connected east and west coast DCs, offices, etc).
And maybe 2 SWEs who handled things like our host OS install system, etc. Basically the next layer above the hardware.
So 9 people that were required to run all of that stuff. But really, NetEng ended up spending like 70% of their time on corporate network things because we'd add offices faster than datacenters.
So if we focused on production only, we really needed about 6 or 7 people total.
I did the math a few times (every single year) and compared our costs, including people, to the costs of moving to AWS 3 year reserve instances.
Doing it ourselves was always half the price.
Of course the difference here was we built on-prem from the start. So there was no repatriation that had to happen.
Since then, I've been responsible for large cloud infra on all 3 major providers and learned a lot about what kind of discounts you can get when you're in the double digit millions in annual spend.
I still think, at a scale of single digit thousands of servers, you'd be cheaper on-prem, fully loaded. But admittedly, I haven't run the numbers since 2017.
- hibikir 5y agoThe problem is not pure cost with a competent team: It's that many companies can't even tell that the team they have is competent. And when the team is not competent, prices, and opportunity costs, are so much higher than what you describe. Imagine a world where adding the equivalent of a 4 core VM with 8 gigs of ram is going to take 6 months from the time a team say they need it, until it's actually available. The meetings, the arguments and other minutiae (like trying to get things installed on the server, because of course you need to play telephone games to get a single command run in a server) make the costs of the server look small. And those companies are in situations where either people have no idea that their teams are performing badly, or lack the political will to do something about it. Adding the services in AWS is just easier, and might be worth every penny. When you go to a high performing firm and explain the procedures and decisions of a low performing one, people think you must be lying, but it's true: Broken IT departments are everywhere in enterprise companies, and paying double of what a good team would take to do the right thing is a bargain, given that they can't even get a whiff at a good team.
- Terretta 5y agoGP “kelp” is the Way, but “hibikir” is the Truth.
- badestrand 5y ago> Imagine a world where adding the equivalent of a 4 core VM with 8 gigs of ram is going to take 6 months from the time a team say they need it, until it's actually available If this is the case then the CTO knows that something is very wrong. I don't think your scenario of companies not knowing whether their teams are incompetent makes sense. You can apply this logic applies to all functions.
- Terretta 5y agoThis is the way. But most firms don’t know how to employ a you to show them this way.
- mwcampbell 5y ago> at a scale of single digit thousands of servers On the other hand, at merely single-digit servers, I'm guessing the fixed cost of personnel referenced in the GP comment would dominate.