4 ms·

No. Big data has a formal definition, but one that most people ignore: When the size of the data becomes part of the problem. (An O'Reilly author said this on

by bravura 14y ago

No. Big data has a formal definition, but one that most people ignore: When the size of the data becomes part of the problem. (An O'Reilly author said this once, I forget his name.) For example, physicists in the 80's who had 10s of MBs of data had a big data problem. Nowadays, this typically means out-of-core data sets.

saosebastiao 14y ago
That definition is but one of many supposed formal definitions. And it isn't a particularly good one, because then "Big Data" becomes a definition that can only be quantified on a person-level of granularity. According to the guy with the Marketing degree from Ho-Hum College, anything bigger than can fit in an Excel spreadsheet is "Big Data". Switch to Doug Cutting, and "Big Data" is in the hundreds of Petabytes. I don't buy the out-of-core definition either. I work using a Data Warehouse with petabytes of information...it certainly doesn't fit in memory. I don't use Hive, Pig, Cascading, etc...(okay, sometimes I use Cascalog, but not as a strategy for dealing with large amounts of data). I use SQL. And it works perfectly fine. But if you ask any of the people out there talking about "Big Data", an SQL database doesn't fit into the definition. Hell, I have done processing on a 200GB CSV file using nothing more than GAWK. Nobody is calling GAWK a big data tool. Face it. "Big Data" is a buzzword for CIOs that read magazines for CIOs but still need to find an engineer to set up their email on their iPad.
- EwanToo 14y ago
  Precisely, the term big data is about selling stuff to CIOs Nobody at yahoo went "we need hadoop to deal with our Big Data problem", it was simply a very large amount of data with relatively limited budget problem, and plenty of very large companies are happy using teradata or netezza to manage PB of information. The new set of tools are often brilliant, but the problems that they solve are almost all not new.

by bravura 14y ago

saosebastiao 14y ago

That definition is but one of many supposed formal definitions. And it isn't a particularly good one, because then "Big Data" becomes a definition that can only be quantified on a person-level of granularity. According to the guy with the Marketing degree from Ho-Hum College, anything bigger than can fit in an Excel spreadsheet is "Big Data". Switch to Doug Cutting, and "Big Data" is in the hundreds of Petabytes. I don't buy the out-of-core definition either. I work using a Data Warehouse with petabytes of information...it certainly doesn't fit in memory. I don't use Hive, Pig, Cascading, etc...(okay, sometimes I use Cascalog, but not as a strategy for dealing with large amounts of data). I use SQL. And it works perfectly fine. But if you ask any of the people out there talking about "Big Data", an SQL database doesn't fit into the definition. Hell, I have done processing on a 200GB CSV file using nothing more than GAWK. Nobody is calling GAWK a big data tool. Face it. "Big Data" is a buzzword for CIOs that read magazines for CIOs but still need to find an engineer to set up their email on their iPad.

EwanToo 14y ago
Precisely, the term big data is about selling stuff to CIOs Nobody at yahoo went "we need hadoop to deal with our Big Data problem", it was simply a very large amount of data with relatively limited budget problem, and plenty of very large companies are happy using teradata or netezza to manage PB of information. The new set of tools are often brilliant, but the problems that they solve are almost all not new.