6 ms·
Something does not add up about hive: They say it has 300 PB, and it generates 4PB per day - which means, at this rate, all data was generated within the last 7
by beagle3 12y ago
Something does not add up about hive: They say it has 300 PB, and it generates 4PB per day - which means, at this rate, all data was generated within the last 75 days.
- boomzilla 12y agoMost likely that the 300PB are distilled/normalized/compacted data whereas the 4PB per day are raw logs.
- bbillings 12y agoThis is the correct answer. Source: I am a FB employee.
- beagle3 12y agoThanks. So, how much does the 4PB of data go down to when it goes into hive? My guess would be something like 200TB. Is it zipped (or lz4/lzo/zopfli/lzma whatever)? Or is it just "distilled"?
- valarauca1 12y ago99% of raw data is useless. Just as a rule. Most of it likely gets tossed into a program to determine if somebody actually needs to do anything, or if something is actually breaking. For example raw user interaction doesn't really grow. While the event is likely ~1kb or so of raw data, at the end your just incrementing a 64bit counter. This is a baseless exaggerated post but should shed some light on.
- beagle3 12y ago> 99% of raw data is useless. Just as a rule. Well, as a non-facebook user, I think 99.9999% of facebook data is useless :) But facebook is in the habit of tracking everyone's surfing habits across the web (through "like" links), and I assume they do more than just "increment a counter with it", even if they don't keep every single detail.
- zeroonetwothree 12y agoData is not stored forever. If most data is only stored ~30 days, then the numbers make sense.
- alkonaut 12y agoAlso 800.000 tables? Surely not in the sense I'm used to, that is, normalized forms where a table corresponds roughy to some business object/noun? Does table mean something else or are there 800k different types of data in there?
- FLUX-YOU 12y agoYeah that's hard to wrap my head around. We have 2k in a large app and that's daunting. Really curious why that came to be (though that's probably the least interesting thing in this article)