5 ms·
The wild confusion around "big data" stems in part from the fact that many people use it to mean something unrelated to data size. I was recently in a back-and
by dbecker 14y ago
The wild confusion around "big data" stems in part from the fact that many people use it to mean something unrelated to data size.
I was recently in a back-and-forth on twitter about this. Some people argued that "big data" refers to the complexity of the analysis or the value of the insight, rather than the size of the data.
Kaggle CEO Anthony Goldbloom advocated for a definition "too big to fit in an excel spreadsheet."
I advocated for for a definition "large enough that the storage and manipulation becomes part of the challenge (in addition to the analysis).
The phrase has taken on so many definitions as to become meaningless.
- bravura 14y agoI argue for your definition: "Big data is when the size of the data becomes part of the problem." Mike Loukides (O'Reilly) mentioned to me that Roger Magoulas (O'Reilly) was the first he heard using that definition. According to this definition, physicists in the 80's were doing big data.
- xradionut 14y agoPeople in science and commerce know data issues and have been dealing with the practice, analysis and theory for years. O'Reilly's just pimpin' the term "Big Data" to sell books. They did the same with "Web 2.0", Java and the initial internet boom.
- dbecker 14y agoO Reilly may be pimping the term to developers, but that's nothing compared to how Cloudera has pimped "Big Data" to enterprise.
- danso 14y agoYes I think sheer size is actually just a small part of "big data". The number of dimensions/facets to the data is more important...don't know if that's the correct term, but I mean the ability to, given the schema of your vistor records, analyze which browser has the highest change in bounce-rate given a certain kind of design change in your site. The old-fashioned visitor record type...the number of database pings per day...would not have this much dimension for analysis. Another part of "big data": the ability to gather, clean, and organize it in a normalized way.
- 001sky 14y ago"large enough that the [storage] and manipulation becomes part of the challenge" -- Also: [acquisition] (not/just storage). the genius of Google and FB. they [create] massive, usable data sets.
- drcube 14y agoAnd here I thought it had to do with the size of the companies involved. "Big Pharma", "Big Tobacco", "Big Oil", "Big Blue" and so on, and now there's "Big Data". The presumption is that big companies have a lot of data, but not necessarily. They may just have unique data needs.
- mark_integerdsv 14y agoFor me, and from an enterprise poit of view, Big Data has always been data external to the enterprise, i.e.: not stored in the warehouse but rather harvested from the web or other sources.