3 ms·
Yep, in the old days (almost 20 years ago now), I worked at a startup with its own racks. We'd go up there once a month to swap some drives. In my 5 years there
by icedchai 3y ago
Yep, in the old days (almost 20 years ago now), I worked at a startup with its own racks. We'd go up there once a month to swap some drives. In my 5 years there, I had to use remote hands exactly twice to reboot a machine. A few other times, one of the servers froze but we were able to reboot it with ILO. This was 3 racks, about 30 machines spread between them (definitely a bunch of empty space, too.)
- throwaway98158 3y agoHappy you, I was in the same situation but I was in the server room daily. Practically every day something went wrong.
- icedchai 3y agoHow many systems did you have? What sorts of problems?
- jjav 3y agoEither you had thousands of machines (at which point is totally makes sense to hire a dedicated person with all the money you're saving) or the site had environmental problems (dirty power, bad cooling, etc?) causing the issues. Hardware is unbelievably reliable, systems normally run for many years without any attention.
- endisneigh 3y agoI am genuinely curious about both the software and hardware stack if you were there every day due to something going wrong. 1. First of all, a good bare metal setup leaves much of the software fleet management able to be done remotely. So if you were there because of that then clearly this wasn't done right. I'm talking about SSH access at a bare minimum, and ideally out of band access as well. 2. I'd say the things that go wrong in a data center are plentiful, but should be predictable. Power, network or cooling related issues means you picked the wrong site or your vendor screwed you. That leaves us to the actual hardware. Sourcing good hardware is obviously critical. Modern Dell and HP enterprise machines should be giving you very, very reliable hardware assuming the beforementioned concerns are addressed. It is true that disks fail, and sometimes memory, but if you were there literally every day there is just a critical failure in your setup somewhere. Even if you had a data center with literally a thousand machines. An uncorrelated failure happening every day is so unlikely that it's not really worth mentioning. Correlate failures could certainly happen. Bad batch of disks, etc, but still shouldn't result in you being there literally every day. I could imagine a series of bad days though, sure.