3 ms·

One place I worked did a backup power test and when they came back from the diesels to the grid the entire datacentre lost power due to a software bug for about

One place I worked did a backup power test and when they came back from the diesels to the grid the entire datacentre lost power due to a software bug for about 10 seconds. It caused a massive outage. The problem was a lot of the machines pulled their OS image from central storage servers and there was no where near enough IO to load everything and they had to prioritise what to bring up first to lighten the load and stop everything thrashing. It was a complete nightmare even though the front end to take sales were well isolated from the backend. Working out what was most important across an entire corporation took as long as the problem resolving slowly by just bringing things up randomly. Nowadays you would just run multiple datacentres or cloud HA and we have SSDs but I just can't see such an architecture understanding being possible for any reasonably large company. The cost of keeping it and the dependencies up to date would be huge and it would always be out of date. More documentation isn't the solution, its to have multiple sites.

jiggawatts 3y ago
That brings back memories of a similar setup with hundreds of Windows servers booting over the network. We had regular “brownouts” even during the day just because the image streaming servers couldn’t handle the IOPS. Basic maintenance would slow down the servers for ten thousand users and generate support tickets. I jumped up and down and convinced management to buy one of the first enterprise SSDs on the market. It was a PCIe card form factor and cost five digits for a tiny amount of storage. We squeezed in the images using block-level deduplication and clever copy scripts that would run the compaction routine after each file was copied. The difference was staggering. Just two of those cards made hundreds of other servers run like greased lightning. Boot times dropped to single digit seconds instead of minutes. Maintenance changes could be done at any time with zero impact on users. The whole cluster could be rebooted all at once with only a slight slowdown. Fun times.

jiggawatts 3y ago

That brings back memories of a similar setup with hundreds of Windows servers booting over the network. We had regular “brownouts” even during the day just because the image streaming servers couldn’t handle the IOPS. Basic maintenance would slow down the servers for ten thousand users and generate support tickets. I jumped up and down and convinced management to buy one of the first enterprise SSDs on the market. It was a PCIe card form factor and cost five digits for a tiny amount of storage. We squeezed in the images using block-level deduplication and clever copy scripts that would run the compaction routine after each file was copied. The difference was staggering. Just two of those cards made hundreds of other servers run like greased lightning. Boot times dropped to single digit seconds instead of minutes. Maintenance changes could be done at any time with zero impact on users. The whole cluster could be rebooted all at once with only a slight slowdown. Fun times.