3 ms·
Good to hear Google being open about the problem. Couple things: - Doesn't each server has its own UPS? I thought some guy from Google showing the machine de
by ww520 17y ago
Good to hear Google being open about the problem. Couple things:
- Doesn't each server has its own UPS? I thought some guy from Google showing the machine design with the battery pack attached to each machine. Why did a data center power outage crash 25% of the machines at the same time?
- Should the servers be restarted and recovered automatically? It said 25% of servers didn't get the UPS power in time and crashed. That implied power was restored shortly and the servers can just be restarted. The servers are just dumb servers that can be shut down, restarted, and recovered in any time. If enough servers were restarted, then BigTable/DataStore would have enough data nodes to continue. The downtime would be couple minutes to tens of minutes.
- Lack of precise monitoring. The first clue of problem was as drop in traffic and outside discussion group posts. Drop-in traffic can be caused by so many things. Shouldn't there be health check monitors on the BigTable and DataStore clusters? Like if more than 10% (25% in this case) of nodes in the clusters are down, raise an alarm and page someone right the way?
- Were there capacity planning and stress testing to determine what percentage of the capacity that can serve traffic in each DC and each cluster? 25% servers down affecting the whole BigTable cluster sounds too little of safety margin.
I don't envy the on-call staff. The pressure must be tremendous.
- simonw 17y agoI seem to remember the on board batteries only last a few minutes - long enough to fail over to another datacenter, supposedly.