3 ms·

How do you deal with hardware management? It seems that at this scale, hardware does not break often enough to warrant full-time system administrator position.

How do you deal with hardware management? It seems that at this scale, hardware does not break often enough to warrant full-time system administrator position. However when hardware breaks, you need to fix it ASAP, you can't spend time looking for someone on the market for one-off job.

vidarh 3y ago
Firstly, if you need to fix hardware ASAP, you're dangerously underprivisoned. You avoid that either by having excess capacity - it's still cheaper than cloud unless you ridiculously overprovision - or you pick a provider that also offers managed servers or cloud servers, and fall back on using that for elasticity (this, incidentally, makes public cloud even less attractive cost-wise - being able to tie in VMs or managed servers into your infra to handle spikes means you can run your own servers far closer to capacity, and so the cost per unit of compute drops dramatically). But when you do need to, any colo provider offers "remote hands" and 24/7 staffing, so if you want you can usually have your server shipped straight to the data centre and just raise a ticket to have them rack and connect it. I have also been second level support for companies for this - getting people, or companies, on retainer separate from the data centre operator, to handle all your ops needs including going out to the data centre in the middle of the night to fix issues their on-site staff can't, is a service you trivially buy. But any reasonable setup these days will have IPMI and/or network boot, so you can have it slotted in an automatically boot into an installer image providing remote access to complete the setup, as well as to power cycle servers and check console output, so hands-on access is rarely needed beyond the initial racking, pulling and replacing hot-swappable drives, or eventual removal. In practice, last time I had racks where we owned the servers as opposed to managed servers where none of this was our problem (still vastly cheaper than public cloud, though somewhat less flexible if you want it to remain cheap), it took on average a fraction of a day per year per rack time spent dealing with anything lower level than our orchestration layer. I tracked the time, and it remained fairly stable over many years, including in cases where our rack costs were so low that it was justified to keep servers in the racks years past the point where they'd been written off.

vidarh 3y ago

Firstly, if you need to fix hardware ASAP, you're dangerously underprivisoned. You avoid that either by having excess capacity - it's still cheaper than cloud unless you ridiculously overprovision - or you pick a provider that also offers managed servers or cloud servers, and fall back on using that for elasticity (this, incidentally, makes public cloud even less attractive cost-wise - being able to tie in VMs or managed servers into your infra to handle spikes means you can run your own servers far closer to capacity, and so the cost per unit of compute drops dramatically). But when you do need to, any colo provider offers "remote hands" and 24/7 staffing, so if you want you can usually have your server shipped straight to the data centre and just raise a ticket to have them rack and connect it. I have also been second level support for companies for this - getting people, or companies, on retainer separate from the data centre operator, to handle all your ops needs including going out to the data centre in the middle of the night to fix issues their on-site staff can't, is a service you trivially buy. But any reasonable setup these days will have IPMI and/or network boot, so you can have it slotted in an automatically boot into an installer image providing remote access to complete the setup, as well as to power cycle servers and check console output, so hands-on access is rarely needed beyond the initial racking, pulling and replacing hot-swappable drives, or eventual removal. In practice, last time I had racks where we owned the servers as opposed to managed servers where none of this was our problem (still vastly cheaper than public cloud, though somewhat less flexible if you want it to remain cheap), it took on average a fraction of a day per year per rack time spent dealing with anything lower level than our orchestration layer. I tracked the time, and it remained fairly stable over many years, including in cases where our rack costs were so low that it was justified to keep servers in the racks years past the point where they'd been written off.