7 ms·
Amazon’s Glacier secret: BDXL (2014)
- zitterbewegung 10y agoNice investigation and also Facebook has been using 50gb blue rays http://www.businessinsider.com/facebook-uses-10000-blu-rays-for-backup-2014-1 http://www.businessinsider.com/facebook-uses-10000-blu-rays-... and is moving to 300gb http://www.businessinsider.com/ces-2016-facebook-uses-panasonic-freezeray-2016-1 http://www.businessinsider.com/ces-2016-facebook-uses-panaso... .
- digi_owl 10y agoAnd Sony have been pushing a BD based backup/archiving system for business use, iirc. Built around multiple discs in a cartridge.
- pacaro 10y agoAnd, to my knowledge, least one specific principal level engineer worked on both systems
- kakarot 10y agoCould you provide me some insight into why optical storage is a better solution than standard HDDs? Is it just the cost, or is cooling / form-factor a big part of it?
- zitterbewegung 10y agoIt is cost that provides the upside for BluRay than HDDs.
- gambiting 10y agoOptical discs cost cents when purchased in bulk, they are literally just plastic + some coating, no expensive metal or electronic parts are required.
- digi_owl 10y agoAre there packet written? That seems to have been the major stumbling block with higher capacity optical media, that one can't do the drag and drop writes that one have with spinning rust and flash chips.
- lobster_johnson 10y agoBulk transfer. They'd write it to a normal disk first, then slowly copy the data in bulk.
- tyingq 10y agoPrevious HN discussion about this: https://news.ycombinator.com/item?id=7647571 https://news.ycombinator.com/item?id=7647571
- mixmastamyk 10y agoI've got a USB3 BDXL writer attached at my desk and it is quite handy and not too expensive. I back up my whole data (work) partition to it every so often and occasionally take one over to a relative's house as my own home-grown "glacier" system.
- chrisper 10y agoWould you not be better off to upload it somewhere instead? Assuming your relative lives close enough, a natural disaster would destroy all your backups.
- mixmastamyk 10y agoIt's a huge state and if it fell into the ocean (wink, wink) data would not be a top-five concern of mine. ;) Also, I prefer privacy. To each his own, however.
- chrisper 10y agoRegarding your last point, I am renting my own server in a datacenter on a different continent. So basically I have total control over my data etc.
- rbanffy 10y ago> So basically I have total control over my data etc. How would you stop someone who gains physical access to your server?
- chrisper 10y agoThat is not really something I have to worry about if the server is in the datacenter of a reputable hoster.
- Guest98123 10y agoIf it's backups, you could just encrypt everything prior to uploading.
- hemancuso 10y agoI highly doubt this. Especially given the introduction of the near line tiers. It's probably just very widely striped, price-segmented data. Also, see: https://news.ycombinator.com/item?id=4416065 https://news.ycombinator.com/item?id=4416065
- binaryanomaly 10y agoDoes someone know about Google nearline and coldline storage? Google claims coldline access within miliseconds.
- monocasa 10y ago> Google claims coldline access within miliseconds Well, that basically tells you almost all you need to know. It's disk in JBODs. The only question is SMR vs conventional. Anyone who knows that can't tell you in public.
- fishnchips 10y agoI think it's because even for cold storage one copy is still persisted on disk.
- CydeWeys 10y agoIf that's how Google is implementing it, and the prices are similar, then isn't that some evidence that Amazon might be implementing it the same way?
- _wmd 10y agoAmazon's latency is measured in hours, whatever process they use involves literally cold storage, either disks that are completely switched off, or some tape-like media archive system, but the article makes a good case for why it's almost certainly not tape
- CydeWeys 10y agoIt could still be normal disks, if we're feeling conspiratorial. It's easy to make faster storage slower; just add waits. Or maybe they make it take up to five hours so they can avoid peak traffic times in whatever data center your info happens to be located in.
- Flammy 10y agoI remember reading this back around when it came out. Have there been any new pieces of the puzzle identified (or announcements...) since then?
- UseStrict 10y agoUsing BDXL seems like a pretty good solution. Most of this data is archival and existing data is very unlikely to change. You can use HDD/SSD as a buffer as users upload data, and then optimize the packing to ensure you're using all available space on a disk. Possibly encrypt each user's data block on the disk. The system itself would only need to track metadata (file metadata, cartridge/disk, key). Deleting a file would be deleting the key and marking the file as inactive. Once/if a cartridge is marked as completely deleted, can just recycle it.
- baybal2 10y agoI feel it will go the way of Data8
- Spooky23 10y agoI thought this was debunked at the time? When I was running exchange systems, our biggest challenge was delivering IOPS. We had to use SAN, and wasted significant storage because we'd spend our IOPS budget at 40-60% storage capacity. I figured at their scale they would have similar problems.
- chrisper 10y agoIOPS isn't important for glacier. You just upload to some buffer and the eventually move it to the slow storage. Reading is pretty slow from glacier.
- Spooky23 10y agoSorry I didn't really finish the point. With exchange, we had all of this expensive, reliable SAN storage that would be perfect for a low requirement glacier like solution. Unfortunately, we lacked the ops mojo to pull it off.
- KaiserPro 10y agoArchive is not about iops. its about streaming bandwidth. for example I used to look after a quantum iScaler 24 drive robot, each drive was capable of kicking out ~100 megabytes a second. It was more than capable of saturating a 40 gig pipe. However random IO was shite, it could take up to 20 minutes to get to random file. (Each tape is stored in a caddy of (from memory) 10 tapes, There is contention on the drives, and then spooling to the right place on the tape.) Email is essentially random IO on a long tale. So, unless your users want a 20 minute delay in accessing last year's emails, I doubt its the right fit. The same applies to Optical disk packs (although the spool time is much less.)
- JimmyAustin 10y agoIt makes sense for email when you aren't giving your users access to their old email, but storing it for regulatory compliance purposes.
- fnord123 10y agoThey're almost certainly doing something like Microsoft's Pelican: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/09/pelican-hotstorage2016.pdf https://www.microsoft.com/en-us/research/wp-content/uploads/... The first comment on TFA says as much. Edit: This is the actual Pelican paper: https://www.microsoft.com/en-us/research/wp-content/uploads/2014/10/osdi2014-Pelican.pdf https://www.microsoft.com/en-us/research/wp-content/uploads/...
- match 10y agoYep. This is basically what they do. Source: I was around when they were designing and building Glacier.
- hbk1966 10y agoThis is very cool.
- CydeWeys 10y agoMy quick TL;DR for other readers (please let me know if I'm wrong): The systems described are servers that are over-provisioned with an order of magnitude more hard drives than you would typically see in a standard server. This allows a much higher ratio of total hard drive space to other components and cooling and power requirements, but at the cost of only being able to spin up some small fraction of the hard drives simultaneously. It then becomes a complicated scheduling problem of optimizing which hard drives to spin up and when based on the total workload of all the data you are trying to read and write at a given time. This system makes sense for data that is expected to be accessed on the order of once per year or less.
- kakoni 10y agoSo Pelican fits 1152 WD WD6001F4PZ 6TB disks in 52U (Source: https://www.microsoft.com/en-us/research/video/rethinking-storage-and-networking-in-next-generation-racks/ https://www.microsoft.com/en-us/research/video/rethinking-st...). I wonder if microsoft going to publish specs (like opencompute?) to pelican servers?
- rrggrr 10y agoCustom engineered SSD. Powered off at rest. See: http://www.storagesearch.com/ssd-petabyte.html http://www.storagesearch.com/ssd-petabyte.html
- hedora 10y agoSSD isn't quite ready for backup use cases (and certainly wasn't when Glacier was built a few years ago). However, is 2017, so we can say for sure that the extrapolation to 2016 in the linked article from 2010 was pretty good. It is too optimistic by a factor of ~2 in density, and ~10x in cost, but is spot-on even compared to most predictions from a year or two ago. Since any whacko can claim they made a prediction in 2010, I double checked: http://web.archive.org/web/20100322200343/http://www.storagesearch.com/ssd-petabyte.html http://web.archive.org/web/20100322200343/http://www.storage... Thanks for sharing the link!
- saosebastiao 10y agoThis is an extremely interesting deductive analysis. However, considering it is amazon, there always exists that persistent "other" possibility: they're purposefully taking a loss.
- ghaff 10y agoOr at any rate, pricing based on net present value of archived data given decreasing costs over time for storage. I do find it rather fascinating that AWS has managed to keep the technology used by Glacier, even at a high-level (i.e. disks vs. tape vs. optical), so under wraps. My personal guess is that it's powered-down disk drives on the grounds that's the simplest long-term solution but that's purely a guess.
- WalterBright 10y agoGiven the scale of Glacier, I'm surprised that Amazon is able to keep their underlying storage technology a secret.
- rbanffy 10y agoThis is one reason to assume it's unremarkable. If Amazon were buying and offloading truckloads of BDXL disks and drives, someone would eventually notice. A good explanation is, therefore, that the technology is unremarkable and boring.
- KaiserPro 10y agoFrom what I recall, Writable optical disks have a much shorter life span compared to tape (~15 years vs 75 years) Plus, if I was designing an archival system, it wouldn't be on blueray, unless there was a requirement for magnetic resistance.
- pmorici 10y agoThey make dvds and brds that are supposed to last 1000 years https://en.m.wikipedia.org/wiki/M-DISC https://en.m.wikipedia.org/wiki/M-DISC
- pmlnr 10y agoYou are mistaken here. Tapes, unless stored with extreme precision, usually last ~20 years safely. They tend to have a longer life, but not without a high risk of deterioration. In my experience, written CD and DVD only lasts for <10 years, if you're lucky. However studies show you can get 30-45, even 45+ years out of them. Most Blu-Ray expectancy exceeds this due to the different non organic dye based layering and coating. The one mentioned above, M-Disc was something developed for DARPA and supposed to last 1000 years in theory. See: http://www.mdisc.com/ http://www.mdisc.com/ http://loc.gov/preservation/resources/rt/NIST_LC_OpticalDiscLongevity.pdf http://loc.gov/preservation/resources/rt/NIST_LC_OpticalDisc... http://www.zdnet.com/article/torture-testing-the-1000-year-dvd/ http://www.zdnet.com/article/torture-testing-the-1000-year-d...
- Twirrim 10y agoEx-Glacier engineer... and no I'm not going to tell you what or how it's done. NDAs and all that jazz. These speculation threads always make for fascinating reading for people on the team.
- exawsthrowaway 10y agoGlacier in particular seems to attract the speculative fascination. Do people not realise the name is not in jest, it really is done with graphene-doped room-temperature ice crystals and laser interference lithography?
- Twirrim 10y agoWe used to joke among ourselves that actually it was done using vinyl records. Have you seen how many vinyl records you can fit into a single rack? Added bonus, 9 out of 10 customers actually preferred the feel of their data when it is restored.
- dwyerm 10y agoI liked the story that the truck-mounted Snowball came from Glacier tech. Amazon has been putting data on a truck and then driving it around Virginia. The delay in reading it back is the time it takes for the truck to arrive at a datacenter and plug in. :)
- sintaks 10y agoSmart man. :)
- Nition 10y agoCould it possibly just be compression on tradtional HDDs? - Cheaper storage because data is heavily compressed - Slow retrieval time due to slow decompression
- usefulcat 10y agoI seriously doubt it. If I have many TB of stuff I need to have saved off site, you can bet the very first thing I'm going to do is compress the hell out of it. So they certainly wouldn't be able to count on compressing it further.
- jayonsoftware 10y agoWhat if they are using 2.5 inch 5TB drives like this http://www.theverge.com/circuitbreaker/2016/11/15/13642078/seagate-backup-plus-portable-5tb-hard-drive http://www.theverge.com/circuitbreaker/2016/11/15/13642078/s... I use. They are nice as we can plug them into a 15 port USB stick, they auto power down when not in used. Amazon could have developed a box like what backblaze.com has done.
- gwicke 10y agoI wonder if Amazon is also deduplicating. It seems likely that a share of users would store large media files or software packages without encryption.
- kennethh 10y agoFacebook disclosed how they are archiving photos long term, they manage a 1 to 1.4 ratio with Reed Solomon Redundancy and 8 disk of 14 can fail without loosing data. https://code.facebook.com/posts/1433093613662262/-under-the-hood-facebook-s-cold-storage-system-/ https://code.facebook.com/posts/1433093613662262/-under-the-...
- A_Crazy_Idea 10y agoGlacier data is stored on densely-packed spun-down disks where only a fraction can be on at a time due to heat and vibration constraints. The wait time reflects the scarcity of disk bandwidth due to these constraints. It was initially fake, but it may be increasingly real as Glacier sees heavier loads.
- shiftpgdn 10y agoWhy isn't anyone thinking tapes? You can get LTO 7 tapes for $0.008 per Gigabyte that allow 100-300 writes before the tape should be destroyed. Quantum and HP make monstrous tape libraries that hold 5-10 petabytes per rack. You can also cartridge-ize your library for even more dense storage on a literal warehouse rack somewhere. Tapes also match the slow retrieval speeds as you have to read the data out onto a drive linearly.
- eridius 10y agoAmazon already denied using tapes, which was mentioned in the third paragraph of the article.
- shiftpgdn 10y agoThere isn't a source mentioned for the denial of tapes but this article claims an Amazon insider verified that it is tapes: http://www.theregister.co.uk/2012/10/18/ipexpo_2012/ http://www.theregister.co.uk/2012/10/18/ipexpo_2012/