4 ms·

I found this part interesting: "F4 currently stores over 65PB of logical data and saves over 53PB of storage." It looks like most of the savings here is due to

by 23david 12y ago

I found this part interesting: "F4 currently stores over 65PB of logical data and saves over 53PB of storage." It looks like most of the savings here is due to optimization of the replication level of storage, according to the usage patterns of the data. Also found it interesting that it doesn't talk about CDN usage.

mark_integerdsv 12y ago
What kind of footprint would a system of this nature require I wonder? Geographically speaking.
- vidarh 12y ago
  You can use 1-3 petabyte per rack as a rough yardstick (before you start counting RAID overhead etc.) It can be more/less than this depending on your rack configuration and how big drives you use, and how dense you pack them (e.g. "standard" off the shelf servers typically peak at 24 3.5" drives per 4U, while top-loaded specialised storage enclosures reach about 50 x 3.5" drives per 4U; full racks vary between 42U and around 47U etc.) But you could if you wanted to get this much storage into about a 5m x 5m server room.... Of course you wouldn't, since you'd want multi-datacentre redundancy etc. Often there's also no point using the biggest drives available because you need higher throughput. Or no point packing them as tight as possible because you need more processing power per drive.
nbm 12y ago
What would you have liked to know about CDN usage? (I work on the team that builds and runs the Facebook CDN infrastructure.)
- 23david 12y ago
  If the replication factor for blobs is calculated from the 'hotness' of the data (by looking at the age for example), I'm wondering how CDNs come into play. Increasing the replication factor would allow for faster reads, but there usually wouldn't be a need for that if you offload reads of hot data to a CDN service (could be an internal CDN...).
  - nbm 12y ago
    There is a just-before-storage cache (through which the CDN edge requests) which reduces requests per object, but it does not affect the number of hot objects per disk. With Haystack each object is fully available with sane performance in degraded state. This means 1/3 I/O per disk, whereas F4 only has 1 easily available full copy, which means 100% load on one disk per object stored. Hope that makes sense - let me know if not.
dbarlett 12y ago
"An Analysis of Facebook Photo Caching" [1] discusses CDN usage and complements the original Haystack paper [2] [3] well. [1] http://www.cs.cornell.edu/~qhuang/papers/sosp_fbanalysis.pdf http://www.cs.cornell.edu/~qhuang/papers/sosp_fbanalysis.pdf [2] https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf https://www.usenix.org/legacy/event/osdi10/tech/full_papers/... [3] http://muratbuffalo.blogspot.com/2010/12/finding-needle-in-haystack-facebooks.html http://muratbuffalo.blogspot.com/2010/12/finding-needle-in-h...

by 23david 12y ago

mark_integerdsv 12y ago

What kind of footprint would a system of this nature require I wonder? Geographically speaking.

vidarh 12y ago
You can use 1-3 petabyte per rack as a rough yardstick (before you start counting RAID overhead etc.) It can be more/less than this depending on your rack configuration and how big drives you use, and how dense you pack them (e.g. "standard" off the shelf servers typically peak at 24 3.5" drives per 4U, while top-loaded specialised storage enclosures reach about 50 x 3.5" drives per 4U; full racks vary between 42U and around 47U etc.) But you could if you wanted to get this much storage into about a 5m x 5m server room.... Of course you wouldn't, since you'd want multi-datacentre redundancy etc. Often there's also no point using the biggest drives available because you need higher throughput. Or no point packing them as tight as possible because you need more processing power per drive.

nbm 12y ago

What would you have liked to know about CDN usage? (I work on the team that builds and runs the Facebook CDN infrastructure.)

23david 12y ago
If the replication factor for blobs is calculated from the 'hotness' of the data (by looking at the age for example), I'm wondering how CDNs come into play. Increasing the replication factor would allow for faster reads, but there usually wouldn't be a need for that if you offload reads of hot data to a CDN service (could be an internal CDN...).
- nbm 12y ago
  There is a just-before-storage cache (through which the CDN edge requests) which reduces requests per object, but it does not affect the number of hot objects per disk. With Haystack each object is fully available with sane performance in degraded state. This means 1/3 I/O per disk, whereas F4 only has 1 easily available full copy, which means 100% load on one disk per object stored. Hope that makes sense - let me know if not.

dbarlett 12y ago

"An Analysis of Facebook Photo Caching" [1] discusses CDN usage and complements the original Haystack paper [2] [3] well. [1] http://www.cs.cornell.edu/~qhuang/papers/sosp_fbanalysis.pdf http://www.cs.cornell.edu/~qhuang/papers/sosp_fbanalysis.pdf [2] https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf https://www.usenix.org/legacy/event/osdi10/tech/full_papers/... [3] http://muratbuffalo.blogspot.com/2010/12/finding-needle-in-haystack-facebooks.html http://muratbuffalo.blogspot.com/2010/12/finding-needle-in-h...