We use ELK (Elasticsearch, Logstash, Kibana) quite heavily for this at my company. It works, and it works well. We feed massive amounts of data into it--it's

We use ELK (Elasticsearch, Logstash, Kibana) quite heavily for this at my company. It works, and it works well. We feed massive amounts of data into it--it's not unusual for our indices to grow by 50 GiB in a single day, which is quite a bit for a small company running a niche website. It's expensive. Very expensive. Not just in terms of money, but also time: the countless hours spent maintaining it, upgrading it, expanding it, securing it, and actually using it. And then, of course, there's the cost of actually knowing what's going on within your infrastructure. It's stressful: attacks are happening constantly. Most of them aren't worth investigating; the firewalls do their job and the attacks fail. But which ones are worth worrying about? Which bugs are actually worth patching? Is it really an issue if a small part of our site goes down for a few minutes--especially when it might cost us 80 hours of work to patch? I doubt these problems can be solved by collecting and analyzing more data. Lack of data isn't a problem--there are already plenty of systems for collecting, analyzing, and storing data. The trouble is using that data in a cost-effective manner. Similarly, I have zero faith in anyone's ability to solve these problems by simply throwing more computing power at them, for much the same reason. I really hope that's not what they're planning to do, but this is Google we're talking about, so I'm not holding my breath. Edit: Clarification, wording