3 ms·

"fat data" (just to avoid calling these mere gigabytes "big data" Mere GB's of data is just a normal database. I have a rule that should be more widespread:

"fat data" (just to avoid calling these mere gigabytes "big data" Mere GB's of data is just a normal database. I have a rule that should be more widespread: If you can put the database in RAM on a x86 server, its not "big data" by any stretch. Then beyond that, it becomes more complex, but for starters lets consider whether the indexes fit in RAM. If the indexes/hashes for your data cannot fit in RAM on a commodity x86 then you probably can consider that you have "big data". So, currently its possible to buy supermicro systems that take 6TB of RAM (just a normal QPI link) without getting into any of the exotic SSI systems (like the SGI UV 2000). We should also avoid talking about the physical plant requirements for "big data" as well, since as its possible to put over 350TB of storage in 4U with products from Nexsan, JetStor, etc. That is over 3PB per rack... So, you can call your data set "big data" if the indexes are > 6TB or the actual data set is > 3PB. These numbers will change next year when new machines/storage arrive.

techdragon 12y ago
I find it easier to think about it from a data "usage" angle, "if you can query anything you like while running the 'database' on a single machine and get your answer within 24hrs your data is not 'big'" I used to see 24hr OLAP cube runs and no one ever called that 'big data'. It's entirely a question of scale in my mind, because these days you can buy truly gigantic servers, and they are phenomenally powerful but if your data needs multiple servers dividing the load in order to perform queries in a timely manner then you start talking about big data, it's a question of scale.
deepsun 12y ago
Yes, we call it "fat data" because of different approach to it, "bigdatey". E.g. it's not relational -- we don't JOIN it with anything (we intentionally designed it so that we stay scalable while growing), and it would perfectly fit a NoSQL datastorage (just don't confuse NoSQL with MongoDB :) After working in Google I (with another teammate) kinda "feel" what's big data: it's more about approach, mindset and toolset to work with it. I agree with you and @techdragon that if you can fit the data into one machine it's probably not really big. But one can also work with 1GB of data using bigdata approach, what we call "fat data". When we grow out of a single machine we won't need to rewrite our project. Nevertheless, this all doesn't prevent our sales team to say "big data" and "cloud" in every their phrase :)

techdragon 12y ago

I find it easier to think about it from a data "usage" angle, "if you can query anything you like while running the 'database' on a single machine and get your answer within 24hrs your data is not 'big'" I used to see 24hr OLAP cube runs and no one ever called that 'big data'. It's entirely a question of scale in my mind, because these days you can buy truly gigantic servers, and they are phenomenally powerful but if your data needs multiple servers dividing the load in order to perform queries in a timely manner then you start talking about big data, it's a question of scale.

deepsun 12y ago

Yes, we call it "fat data" because of different approach to it, "bigdatey". E.g. it's not relational -- we don't JOIN it with anything (we intentionally designed it so that we stay scalable while growing), and it would perfectly fit a NoSQL datastorage (just don't confuse NoSQL with MongoDB :) After working in Google I (with another teammate) kinda "feel" what's big data: it's more about approach, mindset and toolset to work with it. I agree with you and @techdragon that if you can fit the data into one machine it's probably not really big. But one can also work with 1GB of data using bigdata approach, what we call "fat data". When we grow out of a single machine we won't need to rewrite our project. Nevertheless, this all doesn't prevent our sales team to say "big data" and "cloud" in every their phrase :)