5 ms·

Facebook's photo storage rewrite

60 points by sr3d 18y ago

thwarted 18y ago
Removed the reliance on expensive proprietary solutions, but how many non-standard custom tools needed to be written (and thus maintained) to verify, backup and access stuff? All that "extra" POSIX stuff could be avoided with a DOS formatted file system, or one of the many filesystems designed for small embedded storage systems (read and append only). Sites with the scalability and size of Facebook or Flickr ("...Facebook's effort to design a more cost-effective and high-performance storage system for their unique needs") may have a use for this, but the majority of smaller sites would most likely be best served by a direct filesystem store which has known semantics, easy access, and proven history. I'd really like to see some performance numbers that shows which parts of their old storage system started breaking down at which levels of scale.
- lyime 18y ago
  If you go though the older presentation. It breaks down at the disk I/O level. Not only that, it seems like one of the main reason was cost. CDNs can get expensive, especially when are the biggest Photo service on the internet.
  - thwarted 18y ago
    I wasn't really including CDN in "storage system", even though it is listed in those graphics.
  - chris 18y ago
    More specifically, it broke down at metadata operations.
- Andys 18y ago
  At the centre of this redesign is the need to save money. Facebook doesn't need to run on the same hardware (NetApp) as banks and telcos. In this type of application, scaling up is done cheapest with custom software on commodity hardware.
  - jrockway 18y ago
    Additionally, the custom software is pretty straightforward. It's certainly simpler than the rest of their app.
xelfer 18y ago
I hope a new image compression algorithm is installed with it, their current one is horrible.
- jrockway 18y ago
  I don't use Facebook, but I think their goal is to minimize storage costs, even if it means over-compressing images. If you want lossless copies of your photos, use Flickr (or a similar service).
wavesplash 18y ago
I can understand disk I/O limits, but I'm not buying the metadata argument on a read-mostly workload. More detail about WAFL (netapp's filesystem) features and design in this preso: http://www.cs.tau.ac.il/~ohadrode/slides/WAFL.pdf http://www.cs.tau.ac.il/~ohadrode/slides/WAFL.pdf Great for corporate documents. Overkill for photo storage.
- Technophilis 18y ago
  But if it is overkill for photo storage why did facebook use that in the first place ?
  - sokoloff 18y ago
    Because early in a high-growth company's life, speed and ease to market is vastly more important than IOPS, $/TB, TB/KVA, or many other measures that matter a lot when you're facebook's current size, and not much at all when you're facebook's size 5 years ago. Back in 2004, buying an appliance that plugs right in and works the way you understand (presenting a Posix filesystem over NFS) means they can get on with the Facebook-ness of Facebook. 20 hours of research and quoting, 2 hours to rack, network and power it up and you're in business. Once you're at 1/2 TB/day (I actually can't believe it's that low unless they're counting the final storage requirement, not what the users are initially sending them), the linear factor dominates the constant factor, and they can now "afford" the luxury of doing a design that's better in $/TB terms.
- pjhyett 18y ago
  On a much smaller scale, the amount of metadata generated from the Git operations we do at GitHub was killing GFS until we instituted a more aggressive strategy to clear it out. I'm not surprised this was an issue for Facebook.

60 points by sr3d 18y ago

Removed the reliance on expensive proprietary solutions, but how many non-standard custom tools needed to be written (and thus maintained) to verify, backup and access stuff? All that "extra" POSIX stuff could be avoided with a DOS formatted file system, or one of the many filesystems designed for small embedded storage systems (read and append only). Sites with the scalability and size of Facebook or Flickr ("...Facebook's effort to design a more cost-effective and high-performance storage system for their unique needs") may have a use for this, but the majority of smaller sites would most likely be best served by a direct filesystem store which has known semantics, easy access, and proven history. I'd really like to see some performance numbers that shows which parts of their old storage system started breaking down at which levels of scale.

lyime 18y ago
If you go though the older presentation. It breaks down at the disk I/O level. Not only that, it seems like one of the main reason was cost. CDNs can get expensive, especially when are the biggest Photo service on the internet.
- thwarted 18y ago
  I wasn't really including CDN in "storage system", even though it is listed in those graphics.
- chris 18y ago
  More specifically, it broke down at metadata operations.
Andys 18y ago
At the centre of this redesign is the need to save money. Facebook doesn't need to run on the same hardware (NetApp) as banks and telcos. In this type of application, scaling up is done cheapest with custom software on commodity hardware.
- jrockway 18y ago
  Additionally, the custom software is pretty straightforward. It's certainly simpler than the rest of their app.

xelfer 18y ago

I hope a new image compression algorithm is installed with it, their current one is horrible.

jrockway 18y ago
I don't use Facebook, but I think their goal is to minimize storage costs, even if it means over-compressing images. If you want lossless copies of your photos, use Flickr (or a similar service).

wavesplash 18y ago

I can understand disk I/O limits, but I'm not buying the metadata argument on a read-mostly workload. More detail about WAFL (netapp's filesystem) features and design in this preso: http://www.cs.tau.ac.il/~ohadrode/slides/WAFL.pdf http://www.cs.tau.ac.il/~ohadrode/slides/WAFL.pdf Great for corporate documents. Overkill for photo storage.

Technophilis 18y ago
But if it is overkill for photo storage why did facebook use that in the first place ?
- sokoloff 18y ago
  Because early in a high-growth company's life, speed and ease to market is vastly more important than IOPS, $/TB, TB/KVA, or many other measures that matter a lot when you're facebook's current size, and not much at all when you're facebook's size 5 years ago. Back in 2004, buying an appliance that plugs right in and works the way you understand (presenting a Posix filesystem over NFS) means they can get on with the Facebook-ness of Facebook. 20 hours of research and quoting, 2 hours to rack, network and power it up and you're in business. Once you're at 1/2 TB/day (I actually can't believe it's that low unless they're counting the final storage requirement, not what the users are initially sending them), the linear factor dominates the constant factor, and they can now "afford" the luxury of doing a design that's better in $/TB terms.
pjhyett 18y ago
On a much smaller scale, the amount of metadata generated from the Git operations we do at GitHub was killing GFS until we instituted a more aggressive strategy to clear it out. I'm not surprised this was an issue for Facebook.