I was hoping for more than speculation. Sadly, this article didn't contain it, but a bit of googling turned up this paper: http://static.usenix.org/events/sysm

I was hoping for more than speculation. Sadly, this article didn't contain it, but a bit of googling turned up this paper: http://static.usenix.org/events/sysml08/tech/full_papers/xu/xu.pdf http://static.usenix.org/events/sysml08/tech/full_papers/xu/... These guys wrote some software that looks through the logs from a Hadoop cluster and identifies patterns of anomalies. If a cluster of a few exceptions tend to be spammed in the logs at the same time, it identifies this as a single problem. If a scary-looking "error message" is actually normal, it says that this is normal. As someone who has spent too much time doing that kind of thing manually, the idea of automating this sounds very appealing.