5 ms·

This is pretty interesting. I mean, it seems pretty intuitive and obvious once it's put out there like that. I currently deal with a few hundred million items t

This is pretty interesting. I mean, it seems pretty intuitive and obvious once it's put out there like that. I currently deal with a few hundred million items that need to be delivered in real time, and we use a somewhat similar structure, although our algorithms for cache invalidation are much less sophisticated. I wonder how much effort it would take (and I can put forth) to improve my own system to be more efficient. In other words, I wonder how much of this efficiency boost is due to FB's abilities (both in people and technology) to scale. The paper seems to imply that it's relatively simple, now that the data has been gathered, but for a one-person team like mine, I wonder what benefits I can take away from this.

nbm 13y ago
I work with the Facebook CDN team and (amongst other things) maintain the data pipelines that log requests, the tailers that fetch/annotate the requests, and populate the Hive tables that we used for this research and other improvements to serving content. There is certainly a lot of infrastructure that this is built on that many small teams don't immediately have access to in other companies. Whether it's self-service hardware provisioning, Scribe logging infrastructure, tailer frameworks with checkpointing and retries (and job systems to schedule them), and large amounts of available space on Hive for experimentation. But most of the software parts are available as open source, so it doesn't need remain unavailable. This is my first team at Facebook that I've been heavily involved in this scale of data capture and analysis, but it only took a few days to get up to speed through a combination of great tools and good documentation. Being able to drop a Python file in a code repo to ensure that some complex data warehousing task takes place every day after that is pretty powerful.

nbm 13y ago

I work with the Facebook CDN team and (amongst other things) maintain the data pipelines that log requests, the tailers that fetch/annotate the requests, and populate the Hive tables that we used for this research and other improvements to serving content. There is certainly a lot of infrastructure that this is built on that many small teams don't immediately have access to in other companies. Whether it's self-service hardware provisioning, Scribe logging infrastructure, tailer frameworks with checkpointing and retries (and job systems to schedule them), and large amounts of available space on Hive for experimentation. But most of the software parts are available as open source, so it doesn't need remain unavailable. This is my first team at Facebook that I've been heavily involved in this scale of data capture and analysis, but it only took a few days to get up to speed through a combination of great tools and good documentation. Being able to drop a Python file in a code repo to ensure that some complex data warehousing task takes place every day after that is pretty powerful.