3 ms·
Absolutely astounding to me, petabytes an hour? That's in the region of a meg to several megs per user per hour looking at their monthly active user figures.
by IanCal 7y ago
Absolutely astounding to me, petabytes an hour? That's in the region of a meg to several megs per user per hour looking at their monthly active user figures.
- jgalt212 7y agoIt's mostly telemetry data. /s or not /s not sure.
- javiermaestro 7y agoMmm it's not only "telemetry data". It's that (e.g. Scuba) and other types of logs, and not only Facebook (e.g. Instagram as well). Basically, everything that needs logging and post-processing by both real-time systems (e.g. Puma) and batch processing (e.g. all of the data that's ingested and sent to the data warehouse) goes through Scribe. (disclaimer: I work in Scribe) Scuba: https://research.fb.com/publications/scuba-diving-into-data-at-facebook/ https://research.fb.com/publications/scuba-diving-into-data-... Puma: https://research.fb.com/publications/realtime-data-processing-at-facebook/ https://research.fb.com/publications/realtime-data-processin...
- dmix 7y agoDoes that include Instagram and WhatsApp? Or just FB branded stuff like Messenger and the network app...
- packetslave 7y agoNot everything that flows through Scribe is tied to an (external) user, though. Tons of internal systems use it as well, notably anything that logs to Scuba (which is pretty much everything at Facebook. Wide-structured system logs are awesome).