3 ms·
> Paying premiums for BigCloud means not paying salaries You need all the same people to manage the same software stack, regardless of whether the VMs sit on h
by jjav 3y ago
> Paying premiums for BigCloud means not paying salaries
You need all the same people to manage the same software stack, regardless of whether the VMs sit on hardware you own or at an AWS rack. Nearly all the work is on the software side.
The physical bit of maintaining the boxes is a minimal percentage of the work. At various startups we usually didn't have anyone hired to do this work because there wasn't enough to justify a person. Most simple changes (like pull and replace a hot-plug drive) can be done by the colo personnel and for larger work someone would drive to the colo maybe once a month. You only start to need dedicated hardware maintenance personnel at a very large scale.
The premium you're paying for cpu and bandwidth at your BigCloud is so enormous that it'll easily pay many more salaries than the people you need even if you reach the scale of needing dedicated hardware people.
- icedchai 3y agoYep, in the old days (almost 20 years ago now), I worked at a startup with its own racks. We'd go up there once a month to swap some drives. In my 5 years there, I had to use remote hands exactly twice to reboot a machine. A few other times, one of the servers froze but we were able to reboot it with ILO. This was 3 racks, about 30 machines spread between them (definitely a bunch of empty space, too.)
- throwaway98158 3y agoHappy you, I was in the same situation but I was in the server room daily. Practically every day something went wrong.
- icedchai 3y agoHow many systems did you have? What sorts of problems?
- jjav 3y agoEither you had thousands of machines (at which point is totally makes sense to hire a dedicated person with all the money you're saving) or the site had environmental problems (dirty power, bad cooling, etc?) causing the issues. Hardware is unbelievably reliable, systems normally run for many years without any attention.
- endisneigh 3y agoI am genuinely curious about both the software and hardware stack if you were there every day due to something going wrong. 1. First of all, a good bare metal setup leaves much of the software fleet management able to be done remotely. So if you were there because of that then clearly this wasn't done right. I'm talking about SSH access at a bare minimum, and ideally out of band access as well. 2. I'd say the things that go wrong in a data center are plentiful, but should be predictable. Power, network or cooling related issues means you picked the wrong site or your vendor screwed you. That leaves us to the actual hardware. Sourcing good hardware is obviously critical. Modern Dell and HP enterprise machines should be giving you very, very reliable hardware assuming the beforementioned concerns are addressed. It is true that disks fail, and sometimes memory, but if you were there literally every day there is just a critical failure in your setup somewhere. Even if you had a data center with literally a thousand machines. An uncorrelated failure happening every day is so unlikely that it's not really worth mentioning. Correlate failures could certainly happen. Bad batch of disks, etc, but still shouldn't result in you being there literally every day. I could imagine a series of bad days though, sure.