6 ms·

My team of three looks after hundreds of bits of diverse kit in dozens of locations across the world. I can’t remember the last time I took a call outside of o

My team of three looks after hundreds of bits of diverse kit in dozens of locations across the world. I can’t remember the last time I took a call outside of office hours, and even in hours it’s very rare. There’s enough resilience built in that any issues can wait until morning The last major outage was in 2017, before we had a third member of the team. I was on the other side of the world installing a new system, the other was on leave. We had a network issue, OSPF melted and knocked out some services, we were down for about half an hour as I rebooted the core switch pair remotely. (We’ve since redesigned so that doesn’t happen) We get paid nowhere near six figures either. Sure you can be ridiculous, I remember one team I worked on that employed a full time unix contractor (on 3 times the staff wage) to look after 6 servers and deploy a tar ball every few months. I replaced him with a small shell script. Another was a DBA looking after a small oracle database (oracle - which of course is that generations “just use amazon”)

walrus01 5y ago
I think a number of people who are taking the position of "but it's so HARD, and so EXPENSIVE!" to own and run bare metal network infrastructure may not have ever seen a proper OOB management setup, with dedicated OOB network, serial consoles on stuff, management routers and switch at site, things like cradlepoint LTE radios stuck to the top of colocation cages, etc. And then basic other things like having remote smart hands ready to go, and common failure items like fans, power supplies, fan trays, hard drives pre-positioned and ready to swap in. With MOPs for swapping them. Stocks of basic things like fiber patch cables, commonly used transceivers, copper patch cables, stored in every cage.
- PragmaticPulp 5y ago
  > I think a number of people who are taking the position of "but it's so HARD, and so EXPENSIVE!" to own and run bare metal network infrastructure may not have ever seen a proper OOB management setup, with dedicated OOB network, serial consoles on stuff, management routers and switch at site, things like cradlepoint LTE radios stuck to the top of colocation cages, etc. Or we have seen all of this and that's exactly why we don't want it. Building a company is hard enough. Adding the overhead of developing, maintaining, recruiting for, and staffing our own datacenter is madness when I can click a few buttons and get the same thing from a cloud provider without hiring anyone extra to manage the datacenter. No one is denying that a proper data center management system can exist. We all know it can exist. The issue is that it's a huge distraction with a lot of potential pitfalls. Your network infrastructure with Cradlepoint LTE radios in the colocation cages sounds great after it works, gets set up, stays documented, and all the bugs have been ironed out. But that's a lot of hidden work that could have been allocated to launching the product faster.
  - walrus01 5y ago
    I think the difference in point of view here, is that from my own perspective, owning and running the bare metal things is a basic core competency of being an ISP. Which is what I do for a living. The infrastructure I've described up thread is the product. If the use case is somebody developing a software product that is a totally other scenario.
    - spookthesunset 5y ago
      For an ISP, you are probably correct.
  - Corrado 5y ago
    ^^^ 1000x One big pain that no one seems to mention are the mundane details that come with running your own DC. Even in our fairly small operation we had half of a person dedicated to maintaining the warranties on the hardware, sourcing replacement hard drives for 10yo servers, disposing of retired hardware, purchasing new hardware, etc. These things take a huge amount of focus and don't contribute to your product at all. The "cloud" has removed all of this busywork and I wouldn't go back to the old way for almost any amount of money.
  - littlestymaar 5y ago
    > Building a company is hard enough. Adding the overhead of developing, maintaining, recruiting for, and staffing our own datacenter is madness when I can click a few buttons and get the same thing from a cloud provider without hiring anyone extra to manage the datacenter. “Building a company is hard enough, adding the overhead of actually building a company is madness” > But that's a lot of hidden work that could have been allocated to launching the product faster. The topic of this thread isn't about a brand new start-up with no resources trying to build their product as much as possible, it's about a company that spends millions in cloud invoices. Building a company using AWS, totally legit. Scaling this usage up to millions of dollars because you don't want to manage the hassle, fair enough it's your money, I don't mind if all your profit goes to Amazon really…
- spookthesunset 5y ago
  Been there, done that. No thanks. Every place that has their own hardware also has a huge bureaucratic process to get more hardware for your project. Not to mention almost always the software stack is old as dirt. For example mongo might be two or three major versions behind current… and IT wants nothing to do with supporting the new version. People move to the cloud to escape their company’s IT process… there might be some unicorn company out there that does infrastructure “right” but I’ve yet to work there.

walrus01 5y ago

I think a number of people who are taking the position of "but it's so HARD, and so EXPENSIVE!" to own and run bare metal network infrastructure may not have ever seen a proper OOB management setup, with dedicated OOB network, serial consoles on stuff, management routers and switch at site, things like cradlepoint LTE radios stuck to the top of colocation cages, etc. And then basic other things like having remote smart hands ready to go, and common failure items like fans, power supplies, fan trays, hard drives pre-positioned and ready to swap in. With MOPs for swapping them. Stocks of basic things like fiber patch cables, commonly used transceivers, copper patch cables, stored in every cage.

PragmaticPulp 5y ago
> I think a number of people who are taking the position of "but it's so HARD, and so EXPENSIVE!" to own and run bare metal network infrastructure may not have ever seen a proper OOB management setup, with dedicated OOB network, serial consoles on stuff, management routers and switch at site, things like cradlepoint LTE radios stuck to the top of colocation cages, etc. Or we have seen all of this and that's exactly why we don't want it. Building a company is hard enough. Adding the overhead of developing, maintaining, recruiting for, and staffing our own datacenter is madness when I can click a few buttons and get the same thing from a cloud provider without hiring anyone extra to manage the datacenter. No one is denying that a proper data center management system can exist. We all know it can exist. The issue is that it's a huge distraction with a lot of potential pitfalls. Your network infrastructure with Cradlepoint LTE radios in the colocation cages sounds great after it works, gets set up, stays documented, and all the bugs have been ironed out. But that's a lot of hidden work that could have been allocated to launching the product faster.
- walrus01 5y ago
  I think the difference in point of view here, is that from my own perspective, owning and running the bare metal things is a basic core competency of being an ISP. Which is what I do for a living. The infrastructure I've described up thread is the product. If the use case is somebody developing a software product that is a totally other scenario.
  - spookthesunset 5y ago
    For an ISP, you are probably correct.
- Corrado 5y ago
  ^^^ 1000x One big pain that no one seems to mention are the mundane details that come with running your own DC. Even in our fairly small operation we had half of a person dedicated to maintaining the warranties on the hardware, sourcing replacement hard drives for 10yo servers, disposing of retired hardware, purchasing new hardware, etc. These things take a huge amount of focus and don't contribute to your product at all. The "cloud" has removed all of this busywork and I wouldn't go back to the old way for almost any amount of money.
- littlestymaar 5y ago
  > Building a company is hard enough. Adding the overhead of developing, maintaining, recruiting for, and staffing our own datacenter is madness when I can click a few buttons and get the same thing from a cloud provider without hiring anyone extra to manage the datacenter. “Building a company is hard enough, adding the overhead of actually building a company is madness” > But that's a lot of hidden work that could have been allocated to launching the product faster. The topic of this thread isn't about a brand new start-up with no resources trying to build their product as much as possible, it's about a company that spends millions in cloud invoices. Building a company using AWS, totally legit. Scaling this usage up to millions of dollars because you don't want to manage the hassle, fair enough it's your money, I don't mind if all your profit goes to Amazon really…
spookthesunset 5y ago
Been there, done that. No thanks. Every place that has their own hardware also has a huge bureaucratic process to get more hardware for your project. Not to mention almost always the software stack is old as dirt. For example mongo might be two or three major versions behind current… and IT wants nothing to do with supporting the new version. People move to the cloud to escape their company’s IT process… there might be some unicorn company out there that does infrastructure “right” but I’ve yet to work there.