10 ms·

Facebook’s Blu-Ray Powered Cold Storage Data Center

40 points by 1SockChuck 11y ago

y0ghur7_xxx 11y ago
All our chat history is there
- CodingGuy 11y ago
  And our cat images! :)
- chinathrow 11y ago
  I wonder if they had the balls to store deleted conversations as well.
hsshah 11y ago
I am glad companies like Facebook is using Blue-Ray. Hopefully this will trigger progress in increasing density of optical discs and/or decreasing cost of high capacity discs and writers for consumer market. Eager to archive my yearly ever increasing personal media (thanks to having a kid) onto a high capacity disc and just store it in the safety deposit box.
- harryf 11y ago
  Awesome that all our dick pics are being stored forever. Future generations should not miss out.
- bitJericho 11y ago
  I don't know what the life of blu-rays are but burned optical cds and dvds have an estimated lifespan of only 10 years. Make sure you continue to keep up your backups! How I do it is I update my backups every few years but I also keep everything on functional and "hot" hard drives at all times (so I can see it's all still there). There's also http://www.mdisc.com/what-is-mdisc/ http://www.mdisc.com/what-is-mdisc/ but I've never used/seen it.
  - dvdkhlng 11y ago
    The common Blu-Ray media (BD-R, not BD-R LTH) discs use a non-organic dye and should be more resilient than DVD-R or CD-R media. According to this source [1] 100-150 years data retention. Edit: note also, BD-R writers immediately read back and check written data, re-writing damaged sectors to a spare area. This in-hardware defect management should make BD-R more resilient against data corruption than DVD-R or CD-R [2]. [1] https://superuser.com/questions/251369/what-is-the-lifespan-of-dvd-and-blu-ray-discs https://superuser.com/questions/251369/what-is-the-lifespan-... [2] http://fy.chalmers.se/~appro/linux/DVD+RW/Blu-ray/ http://fy.chalmers.se/~appro/linux/DVD+RW/Blu-ray/
  - gambiting 11y ago
    CDs and DVDs have horrible shelf life because they all use organic dyes. Some Blu-Rays do too, but you can still buy Blu-Rays with metallic dyes which should work for decades(panasonic BD-Rs are guaranteed for 50 years). In fact, do you remember the very first, early writable CDs? They all were blue on the bottom - that's because they used a metallic dye as well, I imagine 99% of those discs will be perfectly readable today.
  - slantyyz 11y ago
    The main downside is that MDiscs have a max capacity is 25GB on the Blu-ray. I have some myself, but I can't attest to the life span, since I've only had my BDR for about a year now. How does a normal person prove the lifespan claims of an MDiscs other than waiting? I still find myself burning most backups to 50GB disks, and making redundant copies of only my most important stuff to the 25GB MDiscs. Where I live, a 3 pack of 25GB MDiscs runs around $20 vs $70 for a spindle of 25 RiData 50GB discs. That's ~6.66/25GB MDisc vs ~2.80/50GB BDR. Not a huge premium for piece of mind, I guess.
    - hsshah 11y ago
      I see that they have 100GB BDXL MDisc available now. Have to invest into BDXL writer but that's just one time investment.
grinich 11y ago
A few months ago, Frank Frankovsky left Facebook to start a startup based on this tech! He was previously "VP of Hardware Design & Supply Chain Optimization" and has also been super involved in the Open Compute Project. I'm pretty excited to see what they produce, and hope they can ship before someone like Amazon tries to acquire them. This level of cold storage is going to be critical as the world generates exponentially more data every year. He also mentioned this is an open source venture, which is doubly awesome. :) https://www.facebook.com/frankovsky/posts/10203257723264639?stream_ref=10 https://www.facebook.com/frankovsky/posts/10203257723264639?...
- x5n1 11y ago
  There is a reason that deletion was invented. Everything is not meant to be stored. This is a stupid idea whose time I guess has come.
  - grhmc 11y ago
    Go delete all your photos then.
    - x5n1 11y ago
      too late fb has already permanently stored it somewhere my delete button can't reach.
- lucd 11y ago
  FTA: Last month that startup, Optical Archive, was acquired by Sony. http://www.datacenterknowledge.com/archives/2015/05/27/sony-acquires-facebook-born-startup-repurposes-blu-rays-cold-storage/ http://www.datacenterknowledge.com/archives/2015/05/27/sony-...
  - grinich 11y ago
    ...sigh :(
sschueller 11y ago
Isn't the life span of a burned disc quite bad?
- deleted 11y ago
  [deleted]
- rogeryu 11y ago
  Look at mdisc - it should last a lifetime and longer.
- kalleboo 11y ago
  Isn't the life span of a magnetic disc quite bad?
  - slantyyz 11y ago
    I've gone through over a half dozen Seagate 2.5 drives (spindle and hybrid) and one 3.5 external in the past five years, and I'm lucky if they last 2 years. This includes drives that are mostly offline. I've switched to WD, but I haven't had them long enough to say that they're only better. The WD Green in my Time Capsule has lasted a few years though. All these drive failures led me to buy a BD-R burner, which has worked out fairly well, although I would say it's better to buy burning software than using the free software (on Windows at least) if you want to avoid making a >= $3 coaster. I find the 50GB capacity to be fairly easy to work with, since I'm much less likely to have to span files/folders across discs than the 25GB ones.
- sp332 11y ago
  Rewritable discs are bad, but write-once discs aren't. Especially if you keep them cool and dark.
jacquesm 11y ago
I wonder how Facebook is able to implement the EU privacy directives with this technology as their long term storage solution. If a customer requests deletion of their data you're supposed to comply with that, even on back-ups.
- Cthulhu_ 11y ago
  Yet I don't think anyone will ever check that. How are you going to find one picture that should've been deleted in a cold storage system with exabytes of data (trillions(?) of photos)?
- de_dave 11y ago
  Assuming each item is scrambled with a small, unique key (kept on hot storage), just throw away the key?
- junto 11y ago
  In all seriousness, it is more than likely a flag: User.IsDeleted = true Or User.Status = Status.Deleted (or Status.Deactivated) Facebook allow users to deactivate their accounts, which can then be reactivated. Facebook users can also delete their accounts. Facebook say that this is permanent: When you delete an account, it is permanently deleted from Facebook. You should only delete your account if you are certain you never want to reactivate it. Permanently deleted still doesn't mean deleted. It just means that you can't recover it. Facebook have never publicly stated what their deletion process is. Is it logical or physical? Since Facebook keep anonymous profiles, I wouldn't be surprised if deleted accounts just get migrated to anonymous profiles which they can still profit from. That's what I'd do if I was Facebook.
  - xyzzyz 11y ago
    That kind of flag wouldn't be enough to satisfy EU privacy directives, the comment you're replying to is all about.
  - evgen 11y ago
    "Permanently deleted" means exactly what it says. Deactivated means that the account is inactive but the data is still there. When you delete your account the data is gone within 14 days, no options for recovery; it happens in bulk and with a bit of a grace period for 'buyers remorse' so it is somewhere in the tail end of that 14 day period. And yes, Facebook has publicly stated their deletion process on multiple occasions and has explanations of how it works in online help docs and FAQs that you just choose to ignore.
bloat 11y ago
I used to have a job as one of those robotic arms. The datacenter were I worked had lots of shelves full of tapes, and the computer would beep and display a tape number and a drive number. You'd have to jump up, find the tape and put it in the drive. During the day you were running about constantly trying to keep the thing happy. On the night shift you could sit a watch movies and only get up once or twice an hour.
- chinathrow 11y ago
  One of the real robots dropped a tape within a swiss data center on December 24th 2010 and subsequent error states stopped all debit card transaction as a result...
  - chkuendig 11y ago
    on the day everybody was scrambling to get their christmas shopping done. i still remember the chaos (I also had to get some presents...) edit: it was actually in 2000: http://www.boerse-express.com/mobile#/mobile/pages/96409 http://www.boerse-express.com/mobile#/mobile/pages/96409 (german)
    - chinathrow 11y ago
      Thanks, too late to edit the post for this typo. The ATMS were dead too.
therealmarv 11y ago
I wonder how long it takes to put data from that storage online. What happens e.g. if I want to see my 5 old images fullscreen?
- phire 11y ago
  I think they still keep a copy online. But they only have to keep one copy as the cold storage is used as a backup instead.
Fuxy 11y ago
Can I have one of those Blu-Ray storage units searching all my discs would be way more convenient if i didn't have to do it manually :)
giancarlostoro 11y ago
I wonder how projects like Archive.org could benefit from something like this. I know it has it's up and downs, but if you have enough backups it shouldn't be too bad. I'm sure there's tons of untouched content that could easily be archived (an archive inside the archive), but other sites might as well, such as wikipedia. Of course they would need a bit of funding for this somewhat new endeavour. Maybe facebook can 'donate' the implementation to such sites.
mark_l_watson 11y ago
I really like how open Facebook is about their infrastructure, hardware designs, etc. I would have thought that they would be trying something with higher capacity, but off the shelf == lower costs. Interesting that FB needs to handle 900 million photo uploads a day. With higher resolution cameras on smartphones, their storage requirements will keep increasing. My Note 4 takes 4K video and very high resolution photos - FB, Google, etc. can store lower res versions only, but the pressure will be there for higher resolution support, especially with 4K TVs, and other high resolution devices.
samch 11y ago
Just to help add some clarity, this isn't Blu-Ray in the common consumer sense. I've seen some comments on here about the stability of the dyes and capacity. This technology, as mentioned at the end of the article, is Optical Archive storage [1]. The media are in the form of cartridges with 12 discs in each cartridge, and Sony claims that the media is "50 year rated". Sony currently manufactures cartridges with up to 1.5TB of storage[2]. They also make a 7U, 10-cartridge library [3]. So, in a standard rack, you could fit a maximum of 90TB of Optical Archive storage. That's admittedly not a lot given the storage density of hard drives, so I'm guessing that Facebook's decision to invest in this is partly driven by energy savings or other considerations. [1] https://en.wikipedia.org/wiki/Archival_Disc https://en.wikipedia.org/wiki/Archival_Disc [2] http://pro.sony.com/bbsc/ssr/cat-recmedia/cat-oda/ http://pro.sony.com/bbsc/ssr/cat-recmedia/cat-oda/ [3] http://pro.sony.com/bbsc/ssr/cat-datastorage/cat-opticaldiscarchive/ http://pro.sony.com/bbsc/ssr/cat-datastorage/cat-opticaldisc...
cc438 11y ago
So how does this work with the way Facebook delivers content? I understand the advantages of disc storage for a service like Amazon Glacier as the customer expectations are set for significant time and bandwidth constraints but Facebook is supposed to deliver personal data as fast as it can be served up. They've created a "customer" expectation of instant access that can't be provided by an offline disc. It makes for a good solution as a 2nd or 3rd level backup but you'd need some form of RAID to prevent users from experience a lengthy delay every single time a disk fails. That's a lot of money to invest in a novel solution that creates a lengthy interruption in service should it ever be needed. Maybe the power savings are worth it when weighed against the rarity of such use and expected level of revenue lost to user frustration but that's an equation made fragile by so many variables. I could also see it used in conjunction with some amazing use-prediction algorithm to store data that will most likely never be requested. If Facebook truly keeps everything then they have millions of photos, videos, hidden posts, and other digital detritus tied up in dead, inactive, or orphaned repositories. I know half the groups I joined back in high school are now impossible to find but the data they contained is still important for the purpose of linking people and interests together. I also have several friends who have passed away, making their account a kind of digital mausoleum. Their private data will never be accessed again but it must remain in FB's datacenter until someone is given access to the account to delete it. The number of Facebook accounts tied to the deceased is only increasing so these racks of Blu Rays might be considered the first digital cemetery. I don't know how many resources are allocated to this kind of data but this is where cold storage makes the most sense.

40 points by 1SockChuck 11y ago

y0ghur7_xxx 11y ago

All our chat history is there

CodingGuy 11y ago
And our cat images! :)
chinathrow 11y ago
I wonder if they had the balls to store deleted conversations as well.

hsshah 11y ago

I am glad companies like Facebook is using Blue-Ray. Hopefully this will trigger progress in increasing density of optical discs and/or decreasing cost of high capacity discs and writers for consumer market. Eager to archive my yearly ever increasing personal media (thanks to having a kid) onto a high capacity disc and just store it in the safety deposit box.

harryf 11y ago
Awesome that all our dick pics are being stored forever. Future generations should not miss out.
bitJericho 11y ago
I don't know what the life of blu-rays are but burned optical cds and dvds have an estimated lifespan of only 10 years. Make sure you continue to keep up your backups! How I do it is I update my backups every few years but I also keep everything on functional and "hot" hard drives at all times (so I can see it's all still there). There's also http://www.mdisc.com/what-is-mdisc/ http://www.mdisc.com/what-is-mdisc/ but I've never used/seen it.
- dvdkhlng 11y ago
  The common Blu-Ray media (BD-R, not BD-R LTH) discs use a non-organic dye and should be more resilient than DVD-R or CD-R media. According to this source [1] 100-150 years data retention. Edit: note also, BD-R writers immediately read back and check written data, re-writing damaged sectors to a spare area. This in-hardware defect management should make BD-R more resilient against data corruption than DVD-R or CD-R [2]. [1] https://superuser.com/questions/251369/what-is-the-lifespan-of-dvd-and-blu-ray-discs https://superuser.com/questions/251369/what-is-the-lifespan-... [2] http://fy.chalmers.se/~appro/linux/DVD+RW/Blu-ray/ http://fy.chalmers.se/~appro/linux/DVD+RW/Blu-ray/
- gambiting 11y ago
  CDs and DVDs have horrible shelf life because they all use organic dyes. Some Blu-Rays do too, but you can still buy Blu-Rays with metallic dyes which should work for decades(panasonic BD-Rs are guaranteed for 50 years). In fact, do you remember the very first, early writable CDs? They all were blue on the bottom - that's because they used a metallic dye as well, I imagine 99% of those discs will be perfectly readable today.
- slantyyz 11y ago
  The main downside is that MDiscs have a max capacity is 25GB on the Blu-ray. I have some myself, but I can't attest to the life span, since I've only had my BDR for about a year now. How does a normal person prove the lifespan claims of an MDiscs other than waiting? I still find myself burning most backups to 50GB disks, and making redundant copies of only my most important stuff to the 25GB MDiscs. Where I live, a 3 pack of 25GB MDiscs runs around $20 vs $70 for a spindle of 25 RiData 50GB discs. That's ~6.66/25GB MDisc vs ~2.80/50GB BDR. Not a huge premium for piece of mind, I guess.
  - hsshah 11y ago
    I see that they have 100GB BDXL MDisc available now. Have to invest into BDXL writer but that's just one time investment.

grinich 11y ago

A few months ago, Frank Frankovsky left Facebook to start a startup based on this tech! He was previously "VP of Hardware Design & Supply Chain Optimization" and has also been super involved in the Open Compute Project. I'm pretty excited to see what they produce, and hope they can ship before someone like Amazon tries to acquire them. This level of cold storage is going to be critical as the world generates exponentially more data every year. He also mentioned this is an open source venture, which is doubly awesome. :) https://www.facebook.com/frankovsky/posts/10203257723264639?stream_ref=10 https://www.facebook.com/frankovsky/posts/10203257723264639?...

x5n1 11y ago
There is a reason that deletion was invented. Everything is not meant to be stored. This is a stupid idea whose time I guess has come.
- grhmc 11y ago
  Go delete all your photos then.
  - x5n1 11y ago
    too late fb has already permanently stored it somewhere my delete button can't reach.
lucd 11y ago
FTA: Last month that startup, Optical Archive, was acquired by Sony. http://www.datacenterknowledge.com/archives/2015/05/27/sony-acquires-facebook-born-startup-repurposes-blu-rays-cold-storage/ http://www.datacenterknowledge.com/archives/2015/05/27/sony-...
- grinich 11y ago
  ...sigh :(

sschueller 11y ago

Isn't the life span of a burned disc quite bad?

deleted 11y ago
[deleted]
rogeryu 11y ago
Look at mdisc - it should last a lifetime and longer.
kalleboo 11y ago
Isn't the life span of a magnetic disc quite bad?
- slantyyz 11y ago
  I've gone through over a half dozen Seagate 2.5 drives (spindle and hybrid) and one 3.5 external in the past five years, and I'm lucky if they last 2 years. This includes drives that are mostly offline. I've switched to WD, but I haven't had them long enough to say that they're only better. The WD Green in my Time Capsule has lasted a few years though. All these drive failures led me to buy a BD-R burner, which has worked out fairly well, although I would say it's better to buy burning software than using the free software (on Windows at least) if you want to avoid making a >= $3 coaster. I find the 50GB capacity to be fairly easy to work with, since I'm much less likely to have to span files/folders across discs than the 25GB ones.
sp332 11y ago
Rewritable discs are bad, but write-once discs aren't. Especially if you keep them cool and dark.

jacquesm 11y ago

I wonder how Facebook is able to implement the EU privacy directives with this technology as their long term storage solution. If a customer requests deletion of their data you're supposed to comply with that, even on back-ups.

Cthulhu_ 11y ago
Yet I don't think anyone will ever check that. How are you going to find one picture that should've been deleted in a cold storage system with exabytes of data (trillions(?) of photos)?
de_dave 11y ago
Assuming each item is scrambled with a small, unique key (kept on hot storage), just throw away the key?
junto 11y ago
In all seriousness, it is more than likely a flag: User.IsDeleted = true Or User.Status = Status.Deleted (or Status.Deactivated) Facebook allow users to deactivate their accounts, which can then be reactivated. Facebook users can also delete their accounts. Facebook say that this is permanent: When you delete an account, it is permanently deleted from Facebook. You should only delete your account if you are certain you never want to reactivate it. Permanently deleted still doesn't mean deleted. It just means that you can't recover it. Facebook have never publicly stated what their deletion process is. Is it logical or physical? Since Facebook keep anonymous profiles, I wouldn't be surprised if deleted accounts just get migrated to anonymous profiles which they can still profit from. That's what I'd do if I was Facebook.
- xyzzyz 11y ago
  That kind of flag wouldn't be enough to satisfy EU privacy directives, the comment you're replying to is all about.
- evgen 11y ago
  "Permanently deleted" means exactly what it says. Deactivated means that the account is inactive but the data is still there. When you delete your account the data is gone within 14 days, no options for recovery; it happens in bulk and with a bit of a grace period for 'buyers remorse' so it is somewhere in the tail end of that 14 day period. And yes, Facebook has publicly stated their deletion process on multiple occasions and has explanations of how it works in online help docs and FAQs that you just choose to ignore.

bloat 11y ago

I used to have a job as one of those robotic arms. The datacenter were I worked had lots of shelves full of tapes, and the computer would beep and display a tape number and a drive number. You'd have to jump up, find the tape and put it in the drive. During the day you were running about constantly trying to keep the thing happy. On the night shift you could sit a watch movies and only get up once or twice an hour.

chinathrow 11y ago
One of the real robots dropped a tape within a swiss data center on December 24th 2010 and subsequent error states stopped all debit card transaction as a result...
- chkuendig 11y ago
  on the day everybody was scrambling to get their christmas shopping done. i still remember the chaos (I also had to get some presents...) edit: it was actually in 2000: http://www.boerse-express.com/mobile#/mobile/pages/96409 http://www.boerse-express.com/mobile#/mobile/pages/96409 (german)
  - chinathrow 11y ago
    Thanks, too late to edit the post for this typo. The ATMS were dead too.

therealmarv 11y ago

I wonder how long it takes to put data from that storage online. What happens e.g. if I want to see my 5 old images fullscreen?

phire 11y ago
I think they still keep a copy online. But they only have to keep one copy as the cold storage is used as a backup instead.

Fuxy 11y ago

Can I have one of those Blu-Ray storage units searching all my discs would be way more convenient if i didn't have to do it manually :)

giancarlostoro 11y ago

I wonder how projects like Archive.org could benefit from something like this. I know it has it's up and downs, but if you have enough backups it shouldn't be too bad. I'm sure there's tons of untouched content that could easily be archived (an archive inside the archive), but other sites might as well, such as wikipedia. Of course they would need a bit of funding for this somewhat new endeavour. Maybe facebook can 'donate' the implementation to such sites.

mark_l_watson 11y ago

I really like how open Facebook is about their infrastructure, hardware designs, etc. I would have thought that they would be trying something with higher capacity, but off the shelf == lower costs. Interesting that FB needs to handle 900 million photo uploads a day. With higher resolution cameras on smartphones, their storage requirements will keep increasing. My Note 4 takes 4K video and very high resolution photos - FB, Google, etc. can store lower res versions only, but the pressure will be there for higher resolution support, especially with 4K TVs, and other high resolution devices.

samch 11y ago

Just to help add some clarity, this isn't Blu-Ray in the common consumer sense. I've seen some comments on here about the stability of the dyes and capacity. This technology, as mentioned at the end of the article, is Optical Archive storage [1]. The media are in the form of cartridges with 12 discs in each cartridge, and Sony claims that the media is "50 year rated". Sony currently manufactures cartridges with up to 1.5TB of storage[2]. They also make a 7U, 10-cartridge library [3]. So, in a standard rack, you could fit a maximum of 90TB of Optical Archive storage. That's admittedly not a lot given the storage density of hard drives, so I'm guessing that Facebook's decision to invest in this is partly driven by energy savings or other considerations. [1] https://en.wikipedia.org/wiki/Archival_Disc https://en.wikipedia.org/wiki/Archival_Disc [2] http://pro.sony.com/bbsc/ssr/cat-recmedia/cat-oda/ http://pro.sony.com/bbsc/ssr/cat-recmedia/cat-oda/ [3] http://pro.sony.com/bbsc/ssr/cat-datastorage/cat-opticaldiscarchive/ http://pro.sony.com/bbsc/ssr/cat-datastorage/cat-opticaldisc...

cc438 11y ago

So how does this work with the way Facebook delivers content? I understand the advantages of disc storage for a service like Amazon Glacier as the customer expectations are set for significant time and bandwidth constraints but Facebook is supposed to deliver personal data as fast as it can be served up. They've created a "customer" expectation of instant access that can't be provided by an offline disc. It makes for a good solution as a 2nd or 3rd level backup but you'd need some form of RAID to prevent users from experience a lengthy delay every single time a disk fails. That's a lot of money to invest in a novel solution that creates a lengthy interruption in service should it ever be needed. Maybe the power savings are worth it when weighed against the rarity of such use and expected level of revenue lost to user frustration but that's an equation made fragile by so many variables. I could also see it used in conjunction with some amazing use-prediction algorithm to store data that will most likely never be requested. If Facebook truly keeps everything then they have millions of photos, videos, hidden posts, and other digital detritus tied up in dead, inactive, or orphaned repositories. I know half the groups I joined back in high school are now impossible to find but the data they contained is still important for the purpose of linking people and interests together. I also have several friends who have passed away, making their account a kind of digital mausoleum. Their private data will never be accessed again but it must remain in FB's datacenter until someone is given access to the account to delete it. The number of Facebook accounts tied to the deceased is only increasing so these racks of Blu Rays might be considered the first digital cemetery. I don't know how many resources are allocated to this kind of data but this is where cold storage makes the most sense.