32 ms·

"Big Data" has an identity crisis ... everyone you ask has "Big Data" but I've heard it used to describe 200K (yes thousand) rows of data. I've also seen terab

by smoyer 14y ago

"Big Data" has an identity crisis ... everyone you ask has "Big Data" but I've heard it used to describe 200K (yes thousand) rows of data. I've also seen terabytes of data successfully stored in an SQL database. I think "Big Data" needs a divorce. We've always had data that defied structure, but now (with NoSQL, map-reduce, etc) we have a better idea of how to deal with it. We've still got data processing tasks that can easily be parallelized and others that are inherently linear. Amazingly, when you actually have "Big Data", you need to analyze how best you could process it and architect your systems and software based on the facets listed above. In a corporate BI environment, you should know what questions you're trying to answer before you start. Unfortunately, in a research environment you don't always know what you're looking for. We're trying to bring research scientists and computer scientists to optimize how data is analyzed at the Penn State University ... check out @HackingScience on Twitter.

VonGuard 14y ago
Yeah, you're absolutely right. For a lot of people, big data means Hadoop. For others, it means NoSQL. What I really think it means is just having all of your data in one live, accessible and analytics-ready place. That means logs, customer info, geographic info, demographic data, weather patterns, and just about anything you could ever think of. I like to think that big data is more about aggregating the information available in multiple datasets, and then figuring out how you can read those entrails, as it were. Honestly, big data today is mostly about a potential replacement for data warehouses. The real revolution around big data is that we now have a place to put it for analysis. That was a big hole that Hadoop filled. Now, the next 10 years will be spent actually figuring out how to analyze these big pools of data effectively. Frankly, that's the harder task, I think. We now have the ability to ask questions of ALL the data. Too bad we have no clue what the questions should be.

by smoyer 14y ago

VonGuard 14y ago

Yeah, you're absolutely right. For a lot of people, big data means Hadoop. For others, it means NoSQL. What I really think it means is just having all of your data in one live, accessible and analytics-ready place. That means logs, customer info, geographic info, demographic data, weather patterns, and just about anything you could ever think of. I like to think that big data is more about aggregating the information available in multiple datasets, and then figuring out how you can read those entrails, as it were. Honestly, big data today is mostly about a potential replacement for data warehouses. The real revolution around big data is that we now have a place to put it for analysis. That was a big hole that Hadoop filled. Now, the next 10 years will be spent actually figuring out how to analyze these big pools of data effectively. Frankly, that's the harder task, I think. We now have the ability to ask questions of ALL the data. Too bad we have no clue what the questions should be.