3 ms·
Big data is also a moving target. We can look at size and we can look at rate (read or write). One size classification is: data we can store in memory vs. on
by YZF 13y ago
Big data is also a moving target.
We can look at size and we can look at rate (read or write).
One size classification is: data we can store in memory vs. on drive. I wouldn't refer to anything <64GB as "big" data. If you can store it on one machine it probably shouldn't be considered big from a size perspective (so let's say 4TB as some sort of threshold).
More generally, if you need more than a single server to store or do real-time processing of your data I'd say it qualifies as big. Otherwise probably not.
Another factor a lot of people often don't look at is efficiency. Yes, if everything is a JSON string with a lot of static annotations we can make a small amount of data look very big. You need to look at a more information theoretic "size". This often applies to processing... Yes, you can spend many CPU cycles parsing incoming HTTP requests but that doesn't mean the data is inherently big.