3 ms·
3. Hive is Facebook's data warehouse, with 300 petabytes of data in 800,000 tables. Facebook generates 4 new petabyes of data and runs 600,000 queries and 1 mil
by doque 12y ago
3. Hive is Facebook's data warehouse, with 300 petabytes of data in 800,000 tables. Facebook generates 4 new petabyes of data and runs 600,000 queries and 1 million map-reduce jobs per day.
So 4 PB per day, but only 300 PB total?
- ddoolin 12y agoWas wondering the same thing. My guess is that some also gets removed each day as well, but it seems unlikely.
- evgen 12y agoThink of it like monitoring data. You may collect one-second data on 500 counters per system over 1000 systems, but then you will do a weekly or monthly roll-up where you dump some of the granularity to save space, and after a year you have aggregates that are basically daily trend lines. The more you collect smaller percentage you actually keep.