4 ms·

Facebook just published a blog about moving petabytes per hour. I read it, I find the design interesting. I don't need it. But if I do, it's good to know th

Facebook just published a blog about moving petabytes per hour. I read it, I find the design interesting. I don't need it. But if I do, it's good to know that it's possible, that someone has solved, to have a reference and to possibly use their tool if it's free. So relax, I for one welcome whatever folks are working on out there. If it's true, we can learn from it and apply it in order domains too.

ignoramous 7y ago
> Facebook just published a blog about moving petabytes per hour. For the curious: https://engineering.fb.com/data-infrastructure/scribe/ https://engineering.fb.com/data-infrastructure/scribe/ Edit: HN thread: https://news.ycombinator.com/item?id=21181982 https://news.ycombinator.com/item?id=21181982
- jacquesm 7y ago
  You should post that.
- IanCal 7y ago
  Absolutely astounding to me, petabytes an hour? That's in the region of a meg to several megs per user per hour looking at their monthly active user figures.
  - jgalt212 7y ago
    It's mostly telemetry data. /s or not /s not sure.
    - javiermaestro 7y ago
      Mmm it's not only "telemetry data". It's that (e.g. Scuba) and other types of logs, and not only Facebook (e.g. Instagram as well). Basically, everything that needs logging and post-processing by both real-time systems (e.g. Puma) and batch processing (e.g. all of the data that's ingested and sent to the data warehouse) goes through Scribe. (disclaimer: I work in Scribe) Scuba: https://research.fb.com/publications/scuba-diving-into-data-at-facebook/ https://research.fb.com/publications/scuba-diving-into-data-... Puma: https://research.fb.com/publications/realtime-data-processing-at-facebook/ https://research.fb.com/publications/realtime-data-processin...
  - dmix 7y ago
    Does that include Instagram and WhatsApp? Or just FB branded stuff like Messenger and the network app...
  - packetslave 7y ago
    Not everything that flows through Scribe is tied to an (external) user, though. Tons of internal systems use it as well, notably anything that logs to Scuba (which is pretty much everything at Facebook. Wide-structured system logs are awesome).
rhizome 7y ago
>Facebook just published a blog about moving petabytes per hour. But it's not like that's through one pipe. We don't talk about how many zillions of tons of steel per minute are moved on freeways.

ignoramous 7y ago

> Facebook just published a blog about moving petabytes per hour. For the curious: https://engineering.fb.com/data-infrastructure/scribe/ https://engineering.fb.com/data-infrastructure/scribe/ Edit: HN thread: https://news.ycombinator.com/item?id=21181982 https://news.ycombinator.com/item?id=21181982

jacquesm 7y ago
You should post that.
IanCal 7y ago
Absolutely astounding to me, petabytes an hour? That's in the region of a meg to several megs per user per hour looking at their monthly active user figures.
- jgalt212 7y ago
  It's mostly telemetry data. /s or not /s not sure.
  - javiermaestro 7y ago
    Mmm it's not only "telemetry data". It's that (e.g. Scuba) and other types of logs, and not only Facebook (e.g. Instagram as well). Basically, everything that needs logging and post-processing by both real-time systems (e.g. Puma) and batch processing (e.g. all of the data that's ingested and sent to the data warehouse) goes through Scribe. (disclaimer: I work in Scribe) Scuba: https://research.fb.com/publications/scuba-diving-into-data-at-facebook/ https://research.fb.com/publications/scuba-diving-into-data-... Puma: https://research.fb.com/publications/realtime-data-processing-at-facebook/ https://research.fb.com/publications/realtime-data-processin...
- dmix 7y ago
  Does that include Instagram and WhatsApp? Or just FB branded stuff like Messenger and the network app...
- packetslave 7y ago
  Not everything that flows through Scribe is tied to an (external) user, though. Tons of internal systems use it as well, notably anything that logs to Scuba (which is pretty much everything at Facebook. Wide-structured system logs are awesome).

rhizome 7y ago

>Facebook just published a blog about moving petabytes per hour. But it's not like that's through one pipe. We don't talk about how many zillions of tons of steel per minute are moved on freeways.