6 ms·
> Some customers already have datasets on the order of a petabyte, or 2^50 bytes. Thus the 64-bit capacity limit of 264 bytes is only 14 doublings away. Moore's
by vbezhenar 9y ago
> Some customers already have datasets on the order of a petabyte, or 2^50 bytes. Thus the 64-bit capacity limit of 264 bytes is only 14 doublings away. Moore's Law
for storage predicts that capacity will continue to double every 9-12 months, which means we'll start to hit the 64-bit limit in about a decade. Storage systems tend to live for several decades, so it would be foolish to create a new one without anticipating the needs that will surely arise within its projected lifetime.
So exactly 14 years passed, does someone have 2^64 bytes for a single ZFS filesystem (or anything close to that)? I don't really feel like storage capacity (or 1/price) doubles every year.
- sliken 9y agoLast I checked the ZFS 128 bit values were not byte offsets, but block numbers. So the justification requires over 2^64 blocks, not 2^64 bytes. I calculated it at some point and it was several years of the entire earths manufacturing capacity for disks in a single filesystem. [added the missed "several years of"]
- dijit 9y agoIf you accept the default 128k ZFS block size then it's 2,361,183,241,434,822,606,848 bytes, or 2,097,152 PB Or, exactly 2 Zettabytes.
- Rapzid 9y agoAlso, the Z in ZFS :D
- marzell 9y agoSo, if they didn't use 128-bit... they'd have to change the name! /s
- erric 9y ago>The ZFS name is registered as a trademark of Oracle Corporation; although it was briefly given the retrofitted expanded name "Zettabyte File System", it is no longer considered an initialism. —Wikipedia
- bringtheaction 9y agoI don't believe Wikipedia is right. Search site:oracle.com for zettabyte on google and observe the results being documents and blogs from a multitude of years. ZFS being an acronym for Zettabyte File System was neither retrofitted nor all that brief in use it seems. Former employees of Sun Microsystems, I know some of you browse HN. Care to shed some light on this?
- Rapzid 9y agoIndeed, LLNL have a humongo Lustre/ZFS cluster and kickstarted the ZFSoL project(mucho gracias to LLNL and Brian Behlendorf) and they refer to it as a Zettabyte File System.
- bcantrill 9y agoI wasn't in the room when it happened, but I was probably among the first dozen or so to hear the name -- and it was presented to me as both at once: "ZFS" was both "the last word in filesystems" and (conveniently) a zettabyte. I remember Bonwick quickly pointing out that yes, a yottabyte was larger but "YFS" was passed over for obvious reasons.
- jchw 9y agoI don't think Moore's Law held up for much since then, but it would've been hard to know for sure. That being said, ZFS filesystems can span multiple disks, since it does volume management of its own. I don't know if that means we're going to see zettabyte volumes any time soon, though. After all, 2^64 bytes wouldn't be enough to overflow: it would need to be 2^64 blocks...
- bhouston 9y agoI think the largest file systems in the world are Amazon's S3, and maybe Glacier. I guess Google's internal system used for YouTube and Gmail probably outrank it but there are no numbers for it. But I doubt that they are in a single address space on a single file system. So basically we moved away from singular large unified file systems and built swarms of little file systems.
- vesinisa 9y agoIndeed. "Cloud" is essentially tons of commodity hardware nodes running a tailored userspace distributed file system application. There now certainly exist cloud services that host well in excess of 2^64 bytes, but they are not hosted on traditional file systems, but an array of those. I can't certainly blame their choice though, and 16-million terabyte hard drive arrays (the limit of 64-bit addressing) on a large mainframe are really just a few quantum leaps away. Heck, I still vividly recall buying a MASSIVE 40 GB HDD in 2000 - it seemed excessive beyond wildest dreams at the time (and was promptly filled to brim at a LAN party.) Since, advances in mass storage tech, most importantly the successful commercialization of perpendicular recording in 2005, mean that nowadays 12 TB drives are commercially viable - roughly 1000x the size of a 40 GB drive. In the next 20 years, petabyte-sized drives might well be available (though almost certainly not based on any form of magnetic recording). Once you hook hundred 50-petabyte drives to a single fileserver, you start closing in on limitations of 64-bit addressing.
- killjoywashere 9y agoIsn't there an XKCD about the limit of flash storage being 3 PB per gallon?
- mewse 9y agoI think you may be referring to an image from this What-If article, where the capacity of a gallon of MicroSD cards is estimated at 1.6PB per gallon: https://what-if.xkcd.com/31/ https://what-if.xkcd.com/31/
- runako 9y agoBackblaze posts some good stuff on HD pricing and reliability. Here's a good chart from their data: https://www.backblaze.com/blog/hard-drive-cost-per-gigabyte/ https://www.backblaze.com/blog/hard-drive-cost-per-gigabyte/ They also link to this great (logarithmic) chart: http://www.mkomo.com/cost-per-gigabyte-update http://www.mkomo.com/cost-per-gigabyte-update TL;DR: price hasn't been dropping by half annually, but prices have come down from ~$0.70/GB to < $0.03/GB now. So roughly a factor of 20 over 13-14 years, somewhere in the neighborhood of 25% improvements compounded annually over that time.
- ChuckMcM 9y ago> So exactly 14 years passed, does someone have 2^64 bytes for a single ZFS filesystem (or anything close to that)? I don't really feel like storage capacity (or 1/price) doubles every year. Its unlikely. There are few opportunities to compile a data set of that size. One might be the NSA collections center in Utah, one might be Google or Microsoft's web cache, and one might be something like the Internet Archive's cache. However the length choice gets 'weird' if you want something larger than 64 bits and less than 128. Your next intermediate choice is 96 bits. That is 64 + 32 (or one additional 32 bit long word. Growing by just 8 means you now have a 9 byte pointer, by 16 gives you a 10 byte pointer. Some architectures are penalized when doing off word alignment, and so on those architectures the structure is padded out to the next word size anyway. I've always found Jef's reasoning in that post amusing but I continue to thing 64 bits would have been fine (just like I think it would have been fine for the Internet v6 work :-). Fortunately time has given us a crap ton of memory for our boxes so the storage penalty isn't too great.
- mjevans 9y agoSince 64 bits can serve /many/ use cases and also matches modern machine words I could see there being either two drivers (one with 64 bit native words and the other with full sized ones) or even a fancier driver that uses different ASM bits to just leave half the word all 0s and ignored. The on-disk format could even remain the same; it wouldn't really matter except for lots of small files and those are already so horrid that ZFS might have some other way of handling them. (Like treating /all/ small file names and data as part of a larger directory file or something.)
- Bud 9y agoMoore's Law actually predicted a doubling every two years (for CPUs), not every year, and even the very optimistic Kryder's Law (often cited as the version of Moore's Law for storage) only claimed an increase of 40% per year. Actual progress has fallen far short of that. The rate for 2010-2014 was only 15%/year. Rates were quicker back around 2000-2005, but that pace has fallen dramatically. https://en.wikipedia.org/wiki/Mark_Kryder#Kryder's_law_projection https://en.wikipedia.org/wiki/Mark_Kryder#Kryder's_law_proje...
- wickawic 9y agoI think it is reasonable to expect storage to jump dramatically at such a time that we begin storing “3D” data. Not that everything in VR will be voxels or anything, but even if the fractal dimension of our future data is 2.5 it would obliterate our current needs for storage. /speculation
- AstralStorm 9y agoPhysics provides us with certain density of information in a volume and that is considering advanced quantum storage. May I remind you that flash, DRAM and SRAM already are 3D structures.
- Sean1708 9y agoI think they meant VR graphics data, not data stored on 3D things.
- mixmastamyk 9y agoWonder what the pace of ssd capacity is?
- gowld 9y ago40%/yr is 100%/2years.
- madaxe_again 9y agoCERN are at >100PB, which is only about 6.6 doublings, which is about right with a doubling every two, not one, years. http://iopscience.iop.org/article/10.1088/1742-6596/664/4/042035/pdf http://iopscience.iop.org/article/10.1088/1742-6596/664/4/04...
- sliken 9y agoRight, but does CERN connect all 100PB to a single machine?
- AstralStorm 9y agoTo a cluster fs which ZFS is not. (And not all of it.)
- tempay 9y agoNo. Data is mostly accessed using XRootD[1] which can be used to data from an (almost) unlimited number of nodes. So far as I know, it's optimised for large files so the number of available inodes is limited by RAM. There is also EOS[2] which is based on XRootD and is planned to replace AFS[3] as a general purpose networked filesystem. [1] http://xrootd.org/ http://xrootd.org/ [2] http://eos.web.cern.ch/ http://eos.web.cern.ch/ [3] http://information-technology.web.cern.ch/services/afs-service http://information-technology.web.cern.ch/services/afs-servi...
- mrb 9y agoThey had to be on the conservative side, so they assumed a doubling every 9-12 months. In reality we have seen one doubling every ~34 months in the last 14 years. In 2005 I bought 250GB HDDs for $110 each. In 2018 you can find 8TB HDDs for $150 each. That's 5 doublings. So with 5 doublings, if we saw datasets on the order of 2^50 back then, we should see datasets on the order of 2^55 today (30 PB). And sure enough, here is a computer with a 30 PB global filesystem: https://www.fujitsu.com/downloads/TC/sc11/k-computer-system-overview-sc11.pdf https://www.fujitsu.com/downloads/TC/sc11/k-computer-system-... So another 9 doublings to go to 2^64 bytes. With one every 34 months, this should happen in 25 years (2043). But again, if you are a filesystem developer you should be on the conservative side and assume it might double every ~12 months.
- bastawhiz 9y agoIt's also worth noting that file system developers should think about their file system fifty to a hundred years out. Flash drives today often still come preformatted with FAT32, twenty two years after its release in 1996. Up until a couple years ago, my dad had an external drive formatted with FAT16--a thirty year old file system. If ZFS lives as long as the FATs have (three decades) even at a doubling every 34 months, that means six remaining doublings before 2035. Not quite 2^64, but that's on the liberal side. Half that (doubling every 17 months) is twelve doublings, which is easily over 2^64.
- ohazi 9y agoIt was probably incorrect for the original author to cite Moore's law. Moore's law only ever really held for transistor density on integrated circuits, not magnetic storage. Magnetic storage capacity always grew more slowly. Flash storage capacity today has more to do with die stacking than transistor density. In any case, Moore's Law is definitely dead now. Like, for real this time.
- Johnny555 9y agoBackblaze says they are using 400 PB of storage[1], or around 2^59 bytes. Of course, it's not one big filesystem. They aren't quite doubling annually, but looks like they are growing around ~1.5X each year, so in another decade, they could be close to the 2^64 limit. [1] https://www.backblaze.com/company/about.html https://www.backblaze.com/company/about.html
- sliken 9y agoRight, but their building block is 48 disks x whatever is best profitable (a mix of 4,6,8,10, and 12TB currently or similar). So each system is 500TB or less, zfs is a filesystem for a single node not a distributed filesystem. Even if we ignore that factor of 800 or so (400PB spread across 500 nodes), they still would need another factor of 128,000 before they would need ZFS's 65th bit. Keep in mind it's 2^128 blocks, not 2^128 bytes. Backblaze would need to consume the entire planets supply of disks for a very long time AND put all those disks connected to a single linux, freebsd, or solaris box. Only then would they need the 65th bit that ZFS has for addressing blocks.
- ryacko 9y agoNot really. Global hard drive production is more than 2^64 bytes or 18 exabytes (SI). https://www.anandtech.com/show/10098/market-views-2015-hard-drive-shipments https://www.anandtech.com/show/10098/market-views-2015-hard-...
- tempay 9y agoFrom that it looks like production is 130 exabytes for Seagate and Western Digital combined. Using the default block size on ZFS the limit is 128 * 1024 * 2^64 = 2 zettabytes. Even allowing for other manufacturers it's an order of magnitude more than global annual production.
- pasbesoin 9y agoAn apparently overlooked aspect of the SpaceX Falcon Heavy launch, is the inclusion of a small quartz disc whose format is expected to scale to something like 260 TB (360?, I forget). And that has a lifespan claimed to potentially extend into the billion plus year range. I don't know about that exact disc/format, but if we and as we are finally able to stably write at such capacities... (whatever technologies end up being used to do so). We're going to run into another "640KB should be enough for anyone", unless we are forward thinking with regard to potential storage format capacities. (And if you think no one will ever be able to consume that much data and detail, just think of all the modeling that will be done and the data behind that. Think of the data being put out by CERN and not just how it will expand but also how people will want to explore lower significance hints, corner cases, and who knows what. Etc.)
- anardil 9y agoA fully decked out Isilon cluster supports 68PB on a single file system. https://www.emc.com/collateral/hardware/white-papers/h10719-isilon-onefs-technical-overview-wp.pdf https://www.emc.com/collateral/hardware/white-papers/h10719-...
- wjkohnen 9y agoZFS is not really a 128 bit filesystem, it is merely a 64 bit filesystem that is "128 bit ready". I.e. the on disk format is 128 bit, the kernel drivers are not. At least, they were not until Oracle bought Sun, that's about the last time I really cared about ZFS.