3 ms·

This is the correct answer. Source: I am a FB employee.

• beagle3 12y ago
  Thanks. So, how much does the 4PB of data go down to when it goes into hive? My guess would be something like 200TB. Is it zipped (or lz4/lzo/zopfli/lzma whatever)? Or is it just "distilled"?
  • valarauca1 12y ago
    99% of raw data is useless. Just as a rule. Most of it likely gets tossed into a program to determine if somebody actually needs to do anything, or if something is actually breaking. For example raw user interaction doesn't really grow. While the event is likely ~1kb or so of raw data, at the end your just incrementing a 64bit counter. This is a baseless exaggerated post but should shed some light on.
    • beagle3 12y ago
      > 99% of raw data is useless. Just as a rule. Well, as a non-facebook user, I think 99.9999% of facebook data is useless :) But facebook is in the habit of tracking everyone's surfing habits across the web (through "like" links), and I assume they do more than just "increment a counter with it", even if they don't keep every single detail.