My favorite litmus test for "Big Data" is https://yourdatafitsinram.net https://yourdatafitsinram.net. If you can fit it into RAM on a single machine, you prob

My favorite litmus test for "Big Data" is https://yourdatafitsinram.net https://yourdatafitsinram.net. If you can fit it into RAM on a single machine, you probably shouldn't be using complex distributed systems for working with it. The developer time spent setting it up and fixing arcane bugs due to the distributed system is likely to cost you more than a single monster server will. If your processes generate large amounts of data rapidly, you can get a decent idea when you should start working on a move to a "big data" solution by looking at your rate of growth. ...though in my experience, stuff like logs can be tarballed and tossed on S3 as insurance just in case you really do need to get an overview of some pattern over the past three years. Mostly only about the past six months of logs are really worth keeping on hand actively, IMO.