4 ms·

The real problem is everybody trying to jump on the Big Data™ train. I work a block from the NCSA and everybody is saying big data, even when it means a few gig

The real problem is everybody trying to jump on the Big Data™ train. I work a block from the NCSA and everybody is saying big data, even when it means a few gigabytes in a traditionally non-data format like images. The molecular dynamics simulations at UIUC don't generate Big Data unless you refer to the log files, which are generated once and of an order smaller than most people see. "What is Big Data?" is frequently answered with "What I do!"

EdwardDiego 12y ago
The only working definition of "Big Data" that I've come across is "Data too large to work with in a traditional RDBMS".
- vladsanchez 12y ago
  Yet based on this definition, what constitutes "BigData" may vary from person to person. The other day I performed a simple aggregate against 100M-record dataset in both Hadoop and Postgres. My company thinks we need a BigData (Hadoop + Hive + else) stack to perform such aggregations, sporadically. The results speak for themselves: - Hadoop : 58 minutes - Postgres : 2 minutes Your mileage may vary...
- slurry 12y ago
  "Big Data is any thing which is crash Excel." - @DEVOPS_BORAT
  - EdwardDiego 12y ago
    Oh, that is fantastic. :D
TallGuyShort 12y ago
I've always thought of "Big Data" as being big in the same sense as "Big Oil". It's not any specific capacity of storage or computation - it's the value being extracted from the data to make companies a powerful influence in their industry. I still think the term is overused as well, but I think the "Big" doesn't have to refer to the dataset itself.
- saryant 12y ago
  This is completely off-topic, but "Big Oil" actually did have a specific historic meaning: it's the descendants of the Seven Sisters, the oil companies party to the Consortium of Iran. Those seven were Anglo-Persian (now BP), Gulf + Standard of California + Texaco (now Chevron), Shell, Standard of New Jersey + Standard of New York (Exxon). By popular convention, Total has sort of slipped into this group despite not being party to the Iran agreement, which obviously undermines the historic meaning of the term.
KaiserPro 12y ago
if its not a petabyte its not even close :)

EdwardDiego 12y ago

The only working definition of "Big Data" that I've come across is "Data too large to work with in a traditional RDBMS".

vladsanchez 12y ago
Yet based on this definition, what constitutes "BigData" may vary from person to person. The other day I performed a simple aggregate against 100M-record dataset in both Hadoop and Postgres. My company thinks we need a BigData (Hadoop + Hive + else) stack to perform such aggregations, sporadically. The results speak for themselves: - Hadoop : 58 minutes - Postgres : 2 minutes Your mileage may vary...
slurry 12y ago
"Big Data is any thing which is crash Excel." - @DEVOPS_BORAT
- EdwardDiego 12y ago
  Oh, that is fantastic. :D

TallGuyShort 12y ago

I've always thought of "Big Data" as being big in the same sense as "Big Oil". It's not any specific capacity of storage or computation - it's the value being extracted from the data to make companies a powerful influence in their industry. I still think the term is overused as well, but I think the "Big" doesn't have to refer to the dataset itself.

saryant 12y ago
This is completely off-topic, but "Big Oil" actually did have a specific historic meaning: it's the descendants of the Seven Sisters, the oil companies party to the Consortium of Iran. Those seven were Anglo-Persian (now BP), Gulf + Standard of California + Texaco (now Chevron), Shell, Standard of New Jersey + Standard of New York (Exxon). By popular convention, Total has sort of slipped into this group despite not being party to the Iran agreement, which obviously undermines the historic meaning of the term.

KaiserPro 12y ago

if its not a petabyte its not even close :)