20 ms·

I might suggest a new definition for "Big Data" - Data, whose size is greater than fits in one machine's memory.

by sargun 11y ago

semi-extrinsic 11y ago
Some propose an even stronger definition: when the indices for accessing said data no longer fits in one machine's memory.
lrei 11y ago
Obligatory half-joke: The most commonly used definition deduced from reading online articles is, in terms of size "More data than naively fits into the memory of my (midrange) laptop using a high overhead platform" or in terms of speed "More data per second than can be handled using the same naive database code we used in the 90s for our website's comment section"
bbrazil 11y ago
The definition I like is that it's when the size of the data becomes a significant challenge to solving your problem. For example 1TB of data won't fit in memory, but if all you need to do is a sequential read in under a day then it's not a problem.
- toast0 11y ago
  1TB /will/ fit in memory, you can get an ec2 instance with 2TB; but your point stands.
dalke 11y ago
100 billion records, 6TB max RAM for http://www8.hp.com/uk/en/products/proliant-servers/product-detail.html?oid=8090149#!tab=specs http://www8.hp.com/uk/en/products/proliant-servers/product-d... => 54 bytes per record. BTW, some previous HN discussions along these lines: "Don't use Hadoop - your data isn't that big " - https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html and https://news.ycombinator.com/item?id=6398650 https://news.ycombinator.com/item?id=6398650 "Your data fits in RAM " - http://yourdatafitsinram.com/ http://yourdatafitsinram.com/ - https://news.ycombinator.com/item?id=9581862 https://news.ycombinator.com/item?id=9581862
hvidgaard 11y ago
When I was taking my CS degree, the definition of big data for data intensive applications was exactly that. You cannot have it all in memory, and will have to use the disk. If it fits in memory, you can honestly apply normal algorithmic analysis and optimize for memory access and cpu cycles. Once it no longer fit in memory, you become severely limited by IO.

I might suggest a new definition for "Big Data" - Data, whose size is greater than fits in one machine's memory.

by sargun 11y ago

I might suggest a new definition for "Big Data" - Data, whose size is greater than fits in one machine's memory.

Some propose an even stronger definition: when the indices for accessing said data no longer fits in one machine's memory.

lrei 11y ago

Obligatory half-joke: The most commonly used definition deduced from reading online articles is, in terms of size "More data than naively fits into the memory of my (midrange) laptop using a high overhead platform" or in terms of speed "More data per second than can be handled using the same naive database code we used in the 90s for our website's comment section"

bbrazil 11y ago

The definition I like is that it's when the size of the data becomes a significant challenge to solving your problem. For example 1TB of data won't fit in memory, but if all you need to do is a sequential read in under a day then it's not a problem.

toast0 11y ago
1TB /will/ fit in memory, you can get an ec2 instance with 2TB; but your point stands.

dalke 11y ago

100 billion records, 6TB max RAM for http://www8.hp.com/uk/en/products/proliant-servers/product-detail.html?oid=8090149#!tab=specs http://www8.hp.com/uk/en/products/proliant-servers/product-d... => 54 bytes per record. BTW, some previous HN discussions along these lines: "Don't use Hadoop - your data isn't that big " - https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html and https://news.ycombinator.com/item?id=6398650 https://news.ycombinator.com/item?id=6398650 "Your data fits in RAM " - http://yourdatafitsinram.com/ http://yourdatafitsinram.com/ - https://news.ycombinator.com/item?id=9581862 https://news.ycombinator.com/item?id=9581862

hvidgaard 11y ago

When I was taking my CS degree, the definition of big data for data intensive applications was exactly that. You cannot have it all in memory, and will have to use the disk. If it fits in memory, you can honestly apply normal algorithmic analysis and optimize for memory access and cpu cycles. Once it no longer fit in memory, you become severely limited by IO.