4 ms·

Slightly off topic, but at what point does "big data" just become data? According to wikipedia (which is in line with how I think about big data) "Big data is

Slightly off topic, but at what point does "big data" just become data? According to wikipedia (which is in line with how I think about big data) "Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. " However, as "big data" becomes more mainstream and more tools/services exist to accommodate data at the peta/exobyte scale does that whole definition stop being relevant for kinds of data we're talking about?

CmonDev 13y ago
It's just a buzzword now. "Let's use cloud to do big data analysis to proactively synergise our core competencies".
- frostmatthew 13y ago
  you forgot "disrupt"
npguy 13y ago
You have to be careful when you question big data. It is not recommended. http://fakevalley.com/tech-executive-fired-for-not-using-big-data-in-three-consecutive-meetings/akevalley.com/tech-executive-fired-for-not-using-big-data-in-three-consecutive-meetings/ http://fakevalley.com/tech-executive-fired-for-not-using-big...
kubiiii 13y ago
I guess so and 16 MB was certainely considered too large and complex to process it with traditionel data processing applications a few decades ago.
Keyframe 13y ago
CmonDev put it nicely. Buzzword. Same when you hear, most of the time, how someone is leveraging Hadoop for whatever. There are various definitions of what is big data. I like to think of it as whenever the only way to deal with your data is by doing full table scans, then we're in same domain of thinking of big data.
mcphilip 13y ago
Working in the risk analytics space, "Big Data" seems to be marketing speak for "you can dump all sorts of loosely structured data into this big bucket and our tools will help you find meaningful trends in it." I've yet to see an installation approaching anything near 300GB, so I think of big data as the new sexier label to put on ad hoc data mining applications.
- jeltz 13y ago
  But if it is for running ad hoc data mining on relatively small amounts of data, wouldn't a traditional SQL database do the job just as well? Especially with with features like the JSON support in PostgreSQL.
  - gaius 13y ago
    Yes, you are perfectly correct. But a lot of people are paid a lot of money for reinventing the wheel every few years, so expect your words to fall on deaf ears.
  - jimbokun 13y ago
    I think the question is how much effort will it take to get all kinds of data from all kinds of sources into a nice, coherent schema. The "schema-less" aspects of NoSQL data stores is a big part of the marketing appeal, I think. (Maybe the JSON support in PostgreSQL makes this moot, but that's the thinking.)
    - gaius 13y ago
      You always have a schema. The only question is if you know what it is, or not. In the next few years there will be a lot of money to be made getting data out of these stores and back into relational databases.
  - threeseed 13y ago
    PostgreSQL has terrible sharding and clustering support. So no I would not be using it for any big data projects.
TallGuyShort 13y ago
I actually first heard the term used in what has turned out be to be a very unusual sense. "Big Data" is like "Big Oil" - it's a term for industries that get their economic clout from the value of their data. Google, for instance, is "Big Data" because it's able to offer the advertising platforms and spam filtering services that it does because of the data it's extracting value from.
deleted 13y ago
[deleted]
cbsmith 13y ago
One rule of thumb I've had for people is, "if you can fit it in one computer, it is definitely not big data".
- wjnc 13y ago
  My company actually bought some "big data"-machinery plus consulting and they actually delivered their hardware in a Very Large case. Truly one of those '90s dual big towers. I thought that was pretty funny, considering they put less data in it than fits on a $200 hard drive.

CmonDev 13y ago

It's just a buzzword now. "Let's use cloud to do big data analysis to proactively synergise our core competencies".

frostmatthew 13y ago
you forgot "disrupt"

npguy 13y ago

You have to be careful when you question big data. It is not recommended. http://fakevalley.com/tech-executive-fired-for-not-using-big-data-in-three-consecutive-meetings/akevalley.com/tech-executive-fired-for-not-using-big-data-in-three-consecutive-meetings/ http://fakevalley.com/tech-executive-fired-for-not-using-big...

kubiiii 13y ago

I guess so and 16 MB was certainely considered too large and complex to process it with traditionel data processing applications a few decades ago.

Keyframe 13y ago

CmonDev put it nicely. Buzzword. Same when you hear, most of the time, how someone is leveraging Hadoop for whatever. There are various definitions of what is big data. I like to think of it as whenever the only way to deal with your data is by doing full table scans, then we're in same domain of thinking of big data.

mcphilip 13y ago

Working in the risk analytics space, "Big Data" seems to be marketing speak for "you can dump all sorts of loosely structured data into this big bucket and our tools will help you find meaningful trends in it." I've yet to see an installation approaching anything near 300GB, so I think of big data as the new sexier label to put on ad hoc data mining applications.

jeltz 13y ago
But if it is for running ad hoc data mining on relatively small amounts of data, wouldn't a traditional SQL database do the job just as well? Especially with with features like the JSON support in PostgreSQL.
- gaius 13y ago
  Yes, you are perfectly correct. But a lot of people are paid a lot of money for reinventing the wheel every few years, so expect your words to fall on deaf ears.
- jimbokun 13y ago
  I think the question is how much effort will it take to get all kinds of data from all kinds of sources into a nice, coherent schema. The "schema-less" aspects of NoSQL data stores is a big part of the marketing appeal, I think. (Maybe the JSON support in PostgreSQL makes this moot, but that's the thinking.)
  - gaius 13y ago
    You always have a schema. The only question is if you know what it is, or not. In the next few years there will be a lot of money to be made getting data out of these stores and back into relational databases.
- threeseed 13y ago
  PostgreSQL has terrible sharding and clustering support. So no I would not be using it for any big data projects.

TallGuyShort 13y ago

I actually first heard the term used in what has turned out be to be a very unusual sense. "Big Data" is like "Big Oil" - it's a term for industries that get their economic clout from the value of their data. Google, for instance, is "Big Data" because it's able to offer the advertising platforms and spam filtering services that it does because of the data it's extracting value from.

deleted 13y ago

[deleted]

cbsmith 13y ago

One rule of thumb I've had for people is, "if you can fit it in one computer, it is definitely not big data".

wjnc 13y ago
My company actually bought some "big data"-machinery plus consulting and they actually delivered their hardware in a Very Large case. Truly one of those '90s dual big towers. I thought that was pretty funny, considering they put less data in it than fits on a $200 hard drive.