13 ms·
I think I would consider anything 100Tb and up to be big data. There is no big data that is "easily accessible"; that's why it's "big" because it requires extre
by gaius 12y ago
I think I would consider anything 100Tb and up to be big data. There is no big data that is "easily accessible"; that's why it's "big" because it requires extremely powerful hardware and advanced techniques to work on. Otherwise its just "data".
NOTE: there are people in the world who would laugh at my definition and say that big data starts at 1Pb.
- jleahy 12y agoActually you can normally get away without resorting to "big data" techniques at 1PB scale, it's probably around 10PB that you're really forced to start thinking about things like hadoop style map-reduce/rack-locality.
- Demiurge 12y agoProbably depends on what you're doing with the data?
- jleahy 12y agoRight, I'm thinking of the "embarrassingly parallel" sort of problems, where you can shard the data (for example by city, country, company or some other obvious classifier) and give one shard to one server. I'm you were trying to do anything that's O(n.log n) or O(n^2) (think graph processing) then you'll run into trouble at much smaller scales.
- deleted 12y ago[deleted]
- bbrazil 12y agoIn practice it's more a marketing term, and how big is big depends on what the nature of the data is and what you're doing with it. If it fits in RAM on your laptop, it isn't big data. If you can't process/handle it in a reasonable time on a single machine and your methods need to explicitly worry about how to scale to handle the data volumes it probably is "Big Data". Problems that are embarrassingly parallel need far more data before I'd consider them big (I'd be in the >10PB camp), whereas for relational data I'd say >1TB.
- gaius 12y agoI've worked on 50Tb relational databases, I don't consider myself to be a "big data" guy.
- bbrazil 12y agoI've worked on relational databases of similar size. There's two challenges. The first is maintaining the relational model at that scale is quite tricky, tradeoffs need to be made. The second is the systems-level management of that large a deployment requires a bit more than standard configuration management. These days Amazon have Multi-AZ RDS, which should handle the 2nd item.
- IndianAstronaut 12y agoThe problem with databases that are 50tb or more is that you soon run into limits with the relational model. I have been reading up on different modeling techniques for converting relational models into cassandra's column family stores.
- angrybits 12y agoNo, you run into limits with some relational engines. I work in a Teradata shop and we handle relational models of this size just fine.
- deleted 12y ago[deleted]
- bbrazil 12y agoThe issues are a bit more fundamental. You can't practically fit 50TB on one machine and have reasonable performance, that means multiple machines with the data spread across them. There's then two potential issues: 1) You're doing 1-to-1 joins across tables in a query, network latency may be an issue at high query rates 2) You're going 1-to-many or many-to-many joins across tables in a query, the resulting combinatorial explosion of data is too much to handle You want to have your inner loops/joins as deep down in the stack as possible. If you can structure things so all the heavy lifting stays inside one rack/machine/NUMA node/ processor/core you'll be able to scale a good bit further further. Designing things not to require joins, denormalising and putting it in a column store like Cassandra is also a good approach.
- benihana 12y ago>NOTE: there are people in the world who would laugh at my definition and say that big data starts at 1Pb. I commend them for having a larger penis ^H^H^H^H^H^H data stack than you. I thought big data was less about the actual size of the data store and more about where it comes from (typically passive collection from user activity) and how it's accessed (through some kind of large map-reduce style framework) and used (to inform product decisions or learn more about human behavior)?
- gaius 12y agoSo some other definition of "big" than umm "big" then?
- calinet6 12y agoYes, absolutely. When people talk about big data, more often than not it's a measure of complexity and difficulty, not size.
- saalweachter 12y agoYeah, for everyone but physicists it's really "big enough" data: it's a big enough data set that you've started recording things you didn't even try to record. An excellent example was on HN the other day, using the NYC taxi data to determine which drivers are observant Muslims. It's not something anyone set out to record, but the data set has gotten so large that if you turn it sideways and shake, random facts like that fall out.
- gaius 12y agoDo you think that when people make 1000+ table relational databases it's because a) it's fun b) they're stupid or c) because it's modelling something that is inherently complex? Big data is neither big nor particularly complex.
- sp332 12y agoI have seen several news articles using it that way. I think that's going to become an alternate definition soon.
- learnstats2 12y agoThose numbers make me think of "store all the things" rather than useful statistical data. 1PB is arguably enough data to store genetic variation across all human beings.
- gaius 12y agoI'm referring to guys I know who have worked or presently work at CERN.
- deleted 12y ago[deleted]
- IndianAstronaut 12y agoIn addition to CERN, the LSST is expected to generate 15 terabytes per day.
- iskander 12y ago1pb / 100gb / genome = 10,000 genome sequences. And that's just raw data from one platform. If you're interested in e.g. splicing diversity you would want to do long read RNA sequencing. Leaving room for intermediate results (alignments, assemblies) you would only have room for a thousand people.
- learnstats2 12y agoThanks for picking up on this :) As I said, it's arguable. I was working on the principle that the effective population size of humans is 10,000. (And your genome is oversized, no? 3 billion base-pairs is less than 1 Gigabyte)
- sysk 12y agoWhat I really meant by "easily accessible" was "publicly available".
- jghn 12y agoCount me in as someone who says the word better start with at least "peta" It's not just eh size though, IMO it implies a certain dimensionality and/or lack of structure. At work were sittin on several petabytes and I don't view it as Big Data because it's actually pretty simple. We share many of the same problems as Big Data but not all