One issue is that big data is just too big for the small minds that are tasked with gaining insights from it. It is a really weird hysteria that I can't explain

One issue is that big data is just too big for the small minds that are tasked with gaining insights from it. It is a really weird hysteria that I can't explain. Even network engineers are starting to collect every packet and archive it in some kind of distributed data store like HDFS. What insights are they going to gain from it exactly? If the goal is security then work on better network infrastructure and tools, e.g. libressl. Collecting all that data is not going to get you any closer to making better/smarter networks or allow you to fight DDoS attacks any better because the underlying network infrastructure is what makes it possible in the first place.