13 ms·

ZFS 128 bit storage: Are you high? (2004)

248 points by mixologic 9y ago

deleted 9y ago
[deleted]
vbezhenar 9y ago
> Some customers already have datasets on the order of a petabyte, or 2^50 bytes. Thus the 64-bit capacity limit of 264 bytes is only 14 doublings away. Moore's Law for storage predicts that capacity will continue to double every 9-12 months, which means we'll start to hit the 64-bit limit in about a decade. Storage systems tend to live for several decades, so it would be foolish to create a new one without anticipating the needs that will surely arise within its projected lifetime. So exactly 14 years passed, does someone have 2^64 bytes for a single ZFS filesystem (or anything close to that)? I don't really feel like storage capacity (or 1/price) doubles every year.
- sliken 9y ago
  Last I checked the ZFS 128 bit values were not byte offsets, but block numbers. So the justification requires over 2^64 blocks, not 2^64 bytes. I calculated it at some point and it was several years of the entire earths manufacturing capacity for disks in a single filesystem. [added the missed "several years of"]
  - dijit 9y ago
    If you accept the default 128k ZFS block size then it's 2,361,183,241,434,822,606,848 bytes, or 2,097,152 PB Or, exactly 2 Zettabytes.
    - Rapzid 9y ago
      Also, the Z in ZFS :D
      - marzell 9y ago
        So, if they didn't use 128-bit... they'd have to change the name! /s
      - erric 9y ago
        >The ZFS name is registered as a trademark of Oracle Corporation; although it was briefly given the retrofitted expanded name "Zettabyte File System", it is no longer considered an initialism. —Wikipedia
        bringtheaction 9y ago
        I don't believe Wikipedia is right. Search site:oracle.com for zettabyte on google and observe the results being documents and blogs from a multitude of years. ZFS being an acronym for Zettabyte File System was neither retrofitted nor all that brief in use it seems. Former employees of Sun Microsystems, I know some of you browse HN. Care to shed some light on this?
        Rapzid 9y ago
        Indeed, LLNL have a humongo Lustre/ZFS cluster and kickstarted the ZFSoL project(mucho gracias to LLNL and Brian Behlendorf) and they refer to it as a Zettabyte File System.
        bcantrill 9y ago
        I wasn't in the room when it happened, but I was probably among the first dozen or so to hear the name -- and it was presented to me as both at once: "ZFS" was both "the last word in filesystems" and (conveniently) a zettabyte. I remember Bonwick quickly pointing out that yes, a yottabyte was larger but "YFS" was passed over for obvious reasons.
        avar 9y ago
        What's that obvious reason?
- jchw 9y ago
  I don't think Moore's Law held up for much since then, but it would've been hard to know for sure. That being said, ZFS filesystems can span multiple disks, since it does volume management of its own. I don't know if that means we're going to see zettabyte volumes any time soon, though. After all, 2^64 bytes wouldn't be enough to overflow: it would need to be 2^64 blocks...
- bhouston 9y ago
  I think the largest file systems in the world are Amazon's S3, and maybe Glacier. I guess Google's internal system used for YouTube and Gmail probably outrank it but there are no numbers for it. But I doubt that they are in a single address space on a single file system. So basically we moved away from singular large unified file systems and built swarms of little file systems.
  - vesinisa 9y ago
    Indeed. "Cloud" is essentially tons of commodity hardware nodes running a tailored userspace distributed file system application. There now certainly exist cloud services that host well in excess of 2^64 bytes, but they are not hosted on traditional file systems, but an array of those. I can't certainly blame their choice though, and 16-million terabyte hard drive arrays (the limit of 64-bit addressing) on a large mainframe are really just a few quantum leaps away. Heck, I still vividly recall buying a MASSIVE 40 GB HDD in 2000 - it seemed excessive beyond wildest dreams at the time (and was promptly filled to brim at a LAN party.) Since, advances in mass storage tech, most importantly the successful commercialization of perpendicular recording in 2005, mean that nowadays 12 TB drives are commercially viable - roughly 1000x the size of a 40 GB drive. In the next 20 years, petabyte-sized drives might well be available (though almost certainly not based on any form of magnetic recording). Once you hook hundred 50-petabyte drives to a single fileserver, you start closing in on limitations of 64-bit addressing.
    - killjoywashere 9y ago
      Isn't there an XKCD about the limit of flash storage being 3 PB per gallon?
      - mewse 9y ago
        I think you may be referring to an image from this What-If article, where the capacity of a gallon of MicroSD cards is estimated at 1.6PB per gallon: https://what-if.xkcd.com/31/ https://what-if.xkcd.com/31/
- runako 9y ago
  Backblaze posts some good stuff on HD pricing and reliability. Here's a good chart from their data: https://www.backblaze.com/blog/hard-drive-cost-per-gigabyte/ https://www.backblaze.com/blog/hard-drive-cost-per-gigabyte/ They also link to this great (logarithmic) chart: http://www.mkomo.com/cost-per-gigabyte-update http://www.mkomo.com/cost-per-gigabyte-update TL;DR: price hasn't been dropping by half annually, but prices have come down from ~$0.70/GB to < $0.03/GB now. So roughly a factor of 20 over 13-14 years, somewhere in the neighborhood of 25% improvements compounded annually over that time.
- ChuckMcM 9y ago
  > So exactly 14 years passed, does someone have 2^64 bytes for a single ZFS filesystem (or anything close to that)? I don't really feel like storage capacity (or 1/price) doubles every year. Its unlikely. There are few opportunities to compile a data set of that size. One might be the NSA collections center in Utah, one might be Google or Microsoft's web cache, and one might be something like the Internet Archive's cache. However the length choice gets 'weird' if you want something larger than 64 bits and less than 128. Your next intermediate choice is 96 bits. That is 64 + 32 (or one additional 32 bit long word. Growing by just 8 means you now have a 9 byte pointer, by 16 gives you a 10 byte pointer. Some architectures are penalized when doing off word alignment, and so on those architectures the structure is padded out to the next word size anyway. I've always found Jef's reasoning in that post amusing but I continue to thing 64 bits would have been fine (just like I think it would have been fine for the Internet v6 work :-). Fortunately time has given us a crap ton of memory for our boxes so the storage penalty isn't too great.
  - mjevans 9y ago
    Since 64 bits can serve /many/ use cases and also matches modern machine words I could see there being either two drivers (one with 64 bit native words and the other with full sized ones) or even a fancier driver that uses different ASM bits to just leave half the word all 0s and ignored. The on-disk format could even remain the same; it wouldn't really matter except for lots of small files and those are already so horrid that ZFS might have some other way of handling them. (Like treating /all/ small file names and data as part of a larger directory file or something.)
- Bud 9y ago
  Moore's Law actually predicted a doubling every two years (for CPUs), not every year, and even the very optimistic Kryder's Law (often cited as the version of Moore's Law for storage) only claimed an increase of 40% per year. Actual progress has fallen far short of that. The rate for 2010-2014 was only 15%/year. Rates were quicker back around 2000-2005, but that pace has fallen dramatically. https://en.wikipedia.org/wiki/Mark_Kryder#Kryder's_law_projection https://en.wikipedia.org/wiki/Mark_Kryder#Kryder's_law_proje...
  - wickawic 9y ago
    I think it is reasonable to expect storage to jump dramatically at such a time that we begin storing “3D” data. Not that everything in VR will be voxels or anything, but even if the fractal dimension of our future data is 2.5 it would obliterate our current needs for storage. /speculation
    - AstralStorm 9y ago
      Physics provides us with certain density of information in a volume and that is considering advanced quantum storage. May I remind you that flash, DRAM and SRAM already are 3D structures.
      - Sean1708 9y ago
        I think they meant VR graphics data, not data stored on 3D things.
  - mixmastamyk 9y ago
    Wonder what the pace of ssd capacity is?
  - gowld 9y ago
    40%/yr is 100%/2years.
- madaxe_again 9y ago
  CERN are at >100PB, which is only about 6.6 doublings, which is about right with a doubling every two, not one, years. http://iopscience.iop.org/article/10.1088/1742-6596/664/4/042035/pdf http://iopscience.iop.org/article/10.1088/1742-6596/664/4/04...
  - sliken 9y ago
    Right, but does CERN connect all 100PB to a single machine?
    - AstralStorm 9y ago
      To a cluster fs which ZFS is not. (And not all of it.)
    - tempay 9y ago
      No. Data is mostly accessed using XRootD[1] which can be used to data from an (almost) unlimited number of nodes. So far as I know, it's optimised for large files so the number of available inodes is limited by RAM. There is also EOS[2] which is based on XRootD and is planned to replace AFS[3] as a general purpose networked filesystem. [1] http://xrootd.org/ http://xrootd.org/ [2] http://eos.web.cern.ch/ http://eos.web.cern.ch/ [3] http://information-technology.web.cern.ch/services/afs-service http://information-technology.web.cern.ch/services/afs-servi...
- mrb 9y ago
  They had to be on the conservative side, so they assumed a doubling every 9-12 months. In reality we have seen one doubling every ~34 months in the last 14 years. In 2005 I bought 250GB HDDs for $110 each. In 2018 you can find 8TB HDDs for $150 each. That's 5 doublings. So with 5 doublings, if we saw datasets on the order of 2^50 back then, we should see datasets on the order of 2^55 today (30 PB). And sure enough, here is a computer with a 30 PB global filesystem: https://www.fujitsu.com/downloads/TC/sc11/k-computer-system-overview-sc11.pdf https://www.fujitsu.com/downloads/TC/sc11/k-computer-system-... So another 9 doublings to go to 2^64 bytes. With one every 34 months, this should happen in 25 years (2043). But again, if you are a filesystem developer you should be on the conservative side and assume it might double every ~12 months.
  - bastawhiz 9y ago
    It's also worth noting that file system developers should think about their file system fifty to a hundred years out. Flash drives today often still come preformatted with FAT32, twenty two years after its release in 1996. Up until a couple years ago, my dad had an external drive formatted with FAT16--a thirty year old file system. If ZFS lives as long as the FATs have (three decades) even at a doubling every 34 months, that means six remaining doublings before 2035. Not quite 2^64, but that's on the liberal side. Half that (doubling every 17 months) is twelve doublings, which is easily over 2^64.
    - ianai 9y ago
      And let’s not forget all the crazy (better word?) features like independence from indianness. They built the file system with an eye toward versatility.
      - mnw21cam 9y ago
        > indianness That has to be one of the worst typos I have ever seen.
        arghwhat 9y ago
- ohazi 9y ago
  It was probably incorrect for the original author to cite Moore's law. Moore's law only ever really held for transistor density on integrated circuits, not magnetic storage. Magnetic storage capacity always grew more slowly. Flash storage capacity today has more to do with die stacking than transistor density. In any case, Moore's Law is definitely dead now. Like, for real this time.
- Johnny555 9y ago
  Backblaze says they are using 400 PB of storage[1], or around 2^59 bytes. Of course, it's not one big filesystem. They aren't quite doubling annually, but looks like they are growing around ~1.5X each year, so in another decade, they could be close to the 2^64 limit. [1] https://www.backblaze.com/company/about.html https://www.backblaze.com/company/about.html
  - sliken 9y ago
    Right, but their building block is 48 disks x whatever is best profitable (a mix of 4,6,8,10, and 12TB currently or similar). So each system is 500TB or less, zfs is a filesystem for a single node not a distributed filesystem. Even if we ignore that factor of 800 or so (400PB spread across 500 nodes), they still would need another factor of 128,000 before they would need ZFS's 65th bit. Keep in mind it's 2^128 blocks, not 2^128 bytes. Backblaze would need to consume the entire planets supply of disks for a very long time AND put all those disks connected to a single linux, freebsd, or solaris box. Only then would they need the 65th bit that ZFS has for addressing blocks.
    - ryacko 9y ago
      Not really. Global hard drive production is more than 2^64 bytes or 18 exabytes (SI). https://www.anandtech.com/show/10098/market-views-2015-hard-drive-shipments https://www.anandtech.com/show/10098/market-views-2015-hard-...
      - tempay 9y ago
        From that it looks like production is 130 exabytes for Seagate and Western Digital combined. Using the default block size on ZFS the limit is 128 * 1024 * 2^64 = 2 zettabytes. Even allowing for other manufacturers it's an order of magnitude more than global annual production.
- pasbesoin 9y ago
  An apparently overlooked aspect of the SpaceX Falcon Heavy launch, is the inclusion of a small quartz disc whose format is expected to scale to something like 260 TB (360?, I forget). And that has a lifespan claimed to potentially extend into the billion plus year range. I don't know about that exact disc/format, but if we and as we are finally able to stably write at such capacities... (whatever technologies end up being used to do so). We're going to run into another "640KB should be enough for anyone", unless we are forward thinking with regard to potential storage format capacities. (And if you think no one will ever be able to consume that much data and detail, just think of all the modeling that will be done and the data behind that. Think of the data being put out by CERN and not just how it will expand but also how people will want to explore lower significance hints, corner cases, and who knows what. Etc.)
- anardil 9y ago
  A fully decked out Isilon cluster supports 68PB on a single file system. https://www.emc.com/collateral/hardware/white-papers/h10719-isilon-onefs-technical-overview-wp.pdf https://www.emc.com/collateral/hardware/white-papers/h10719-...
- wjkohnen 9y ago
  ZFS is not really a 128 bit filesystem, it is merely a 64 bit filesystem that is "128 bit ready". I.e. the on disk format is 128 bit, the kernel drivers are not. At least, they were not until Oracle bought Sun, that's about the last time I really cared about ZFS.
CognitiveLens 9y ago
> If 64 bits isn't enough, the next logical step is 128 bits. Can someone explain this? Is there some kind of awkwardness/waste with anything less than doubling the number of bits?
- Analemma_ 9y ago
  Having an integer/memory address size that isn't a power of two is possible, and has been done before, but it's awkward. Lacking a specific reason not to go to 128 bits, they did that.
  - sliken 9y ago
    You mean like doubling the overhead of every pointer into the filesystem? Seems directly related to the rather high ram overhead (I've heard 1GB ram per TB of disk).
    - lmm 9y ago
      Pointers in the filesystem would want to be word-aligned anyway, so once you make them larger than 8 bytes you might as well make them 16 bytes.
      - sliken 9y ago
        Right, but the entire point is that 64 bits is PLENTY. The default block size is 128KB. So 2^64 * block size = 2417851639229258349412352 bytes Or 2147483648 PB. Sure there might be distributed systems that are mind boggling large, but those aren't in a single filesystem attached to a single node. ZFS would be a better filesystem with "only" 64 bit blocks. Today's higher density nodes are something like 48 disks * 12TB, so approximately 2 per PB. 12 per rack would be 6PB. So 178,956,970 racks consuming 300 times the annual production on earth and you'd start wishing for that 65th bit. All in a single OS connecting to a single storage system. Or you could assume that in the next few decades you'd be batshit crazy to want to install that much storage on a single zfs system and half the overhead of every pointer into the filesystem.
znpy 9y ago
"we don't want to deal with this shit for another forty years or possibly forever" would have sufficed for me.
- DaiPlusPlus 9y ago
  I’m struggling to think of a project that released a new version to accommodate some far-off limit that wasn’t blindsided and obsoleted by something operating on a different paradigm - like using YYMMDD for dates, the changing it to YYYYMMDD for Y2K when they should have been using integer time_t - but then that was 32-bit and now 64-bit. But we won’t even be 10% of the way there before a relativistic-capable time system will replace it. So why worry? We’re doomed to rewrite it anyway :)
  - regularfry 9y ago
    Trading off a tiny design decision like that against knowing you're unlikely to have to ever revisit except in an unforeseeable situation is a pretty good place to be.
frozenport 9y ago
Why can't they have a 64 bit and 128 bit ZFS offering side-by-side. When somebody actually needs 128bit they can migrate?
chickenthief 9y ago
"Thus, fully populating a 128-bit storage pool would, literally, require more energy than boiling the oceans.". Could we instead boil some Jupiter Moons?
- saalweachter 9y ago
  Boiling the oceans is an alternative energy use, not a source.
paulsutter 9y ago
Real-world large filesystems are distributed across many thousands of hosts and multiple datacenters, not mounted as a Linux filesystem on a single host. Because whole racks and whole datacenters fail, not just disk drives. So they used 128 bit because of bikeshedding. Committees always make the most conservative decision possible. Like IPv6.
- zzzcpan 9y ago
  On the other hand such committee bikeshedding seems to work rather well for PR. It makes them ignore hard problems and instead focus on things most people can understand and relate. Gaining more trust as opposed to a well designed thing with nothing to understand or relate.
- kstrauser 9y ago
  Except IPv6 being 128-bit makes fast hardware implementations much easier than if it had to deal with shorter prefix lengths. Nothing shorter than 128 really makes sense at all in an IPv4 replacement.
  - mixmastamyk 9y ago
    No comprendo. Why would 128 be faster than 64?
    - sliken 9y ago
      IPV6 is basically 64 bits for routing and 64 bits for the local network segment. Seems plausible that it's faster than trying to mask out the bits you need.
      - mixmastamyk 9y ago
        Why not 32:32? Shouldn’t four billion internets be enough?
        kstrauser 9y ago
ktpsns 9y ago
> Thus, fully populating a 128-bit storage pool would, literally, require more energy than boiling the oceans. This is actually an argument against 128 bit, because it clearly shows that 128 bit are unreachable and thus a waste. What about 96 bit?
- DaiPlusPlus 9y ago
  96-bit, and other non-power-of-two systems are cumbersome and error-prone to work with in C - which is often used when writing firmware for computer hardware. The real risks of economic damage caused by bit-fiddling bugs is much greater than the risk of bringing the universe’s thermodynamic heat-death ever so slightly nearer...
  - speleding 9y ago
    I would think it's fairly easy to keep all calculations in C in 128-bit and just mask out the top few bytes when retrieving and storing. You could also argue that >64 bit values will be rare enough that they warrant their own code path if they are encountered as an optimization (perhaps they already do this?).
russellbeattie 9y ago
Back in 2004, they were probably thinking, "Well, IPv6 is using 128bits, and that's going to be the standard any day now..."
NamTaf 9y ago
What's the downside to having 128 bits over 64? I'm not familiar enough with file systems to know, so I'm left wondering if there's any downside to oversizing the hammer for the nail?
- kev009 9y ago
  It expands the data structure size, which multiplies N for both on disk and in RAM representation (and cache and shorn cache lines). Many modern architectures have native 128b types (amd64, POWER) so it's not really a big deal to the CPU itself even if you need to do atomic operations for concurrency. OTOH it guarantees there wont really need to be on disk changes for pointer sizes. That may be useful in situations that weren't really imagined, for instance 128b is enough for a Single Level Store where persistent and working set (or NV) are all in the same mappable space for any conceivable workload.
  - deleted 9y ago
    [deleted]
- dmytrish 9y ago
  tl;dr: doubling size of metadata (and usually a half of a 128bit address is zeros anyway). Having twice as much space occupied has many implications: twice as much space lost for data itself, less metadata will fit CPU caches, less throughput in buses between CPU and storage, lengthier read time for the storage itself, etc.
PeCaN 9y ago
> Thus, fully populating a 128-bit storage pool would, literally, require more energy than boiling the oceans. So we should be good until OceanCoin is introduced.
exabrial 9y ago
Is it true zfs never saw widespread adoption because of licensing compat with Linux and patents or has come and gone?
- boomboomsubban 9y ago
  People are reluctant to use openZFS on Linux as it can't be included in the kernel due to license issues. I'd still say it has widespread adoption when you consider the people willing to run it on Linux and how popular it is on FreeBSD/some other systems.
- kikoreis 9y ago
  FWIW it has been included in stock Ubuntu since 16.04 LTS so it is widely available, and likely more in use than assumed.
- rayvd 9y ago
  I think that certainly hurt it. Also, when Oracle bought Sun, they started restricting Solaris use to Sun hardware only. Previously, we were happily using Solaris 10_x86 + ZFS on SuperMicro boxes. We still have north of 7PB of ZFS-based storage at $DAYJOB (NexentaStor based and Dell hardware) after introducing it nearly eight years ago. ZFS itself has been awesome.
natehouk 9y ago
https://www.sec.gov/whistleblower/ https://www.sec.gov/whistleblower/
z3t4 9y ago
It could run the file system of the death star (Star Wars).
sova 9y ago
>That's enough to survive Moore's Law until I'm dead, and after that, it's not my problem. ...really?
- mort96 9y ago
  No, I don't think that was entirely serious. 128 bytes should be more than enough to last for a very long time, not just a little bit longer than the author's life.
  - mort96 9y ago
    Damn, just noticed I wrote 128 bytes and not 128 bits, too late to edit.
    - sova 9y ago
      I wasn't talking about the number of bits, my friend. I was talking about the attitude that "it's not my problem after I'm dead." That's exactly why we are stuck with so many societal problems. For example, the Federal Reserve Act of 1913 was a true victory for the sons-of-guns who got it enacted, but then they died and now we're all left with the aftermath. It's this shortsighted attitude that is the problem, not the number of bits in the implementation.

248 points by mixologic 9y ago

deleted 9y ago

[deleted]

> Some customers already have datasets on the order of a petabyte, or 2^50 bytes. Thus the 64-bit capacity limit of 264 bytes is only 14 doublings away. Moore's Law for storage predicts that capacity will continue to double every 9-12 months, which means we'll start to hit the 64-bit limit in about a decade. Storage systems tend to live for several decades, so it would be foolish to create a new one without anticipating the needs that will surely arise within its projected lifetime. So exactly 14 years passed, does someone have 2^64 bytes for a single ZFS filesystem (or anything close to that)? I don't really feel like storage capacity (or 1/price) doubles every year.

sliken 9y ago
Last I checked the ZFS 128 bit values were not byte offsets, but block numbers. So the justification requires over 2^64 blocks, not 2^64 bytes. I calculated it at some point and it was several years of the entire earths manufacturing capacity for disks in a single filesystem. [added the missed "several years of"]
- dijit 9y ago
  If you accept the default 128k ZFS block size then it's 2,361,183,241,434,822,606,848 bytes, or 2,097,152 PB Or, exactly 2 Zettabytes.
  - Rapzid 9y ago
    Also, the Z in ZFS :D
    - marzell 9y ago
      So, if they didn't use 128-bit... they'd have to change the name! /s
    - erric 9y ago
      >The ZFS name is registered as a trademark of Oracle Corporation; although it was briefly given the retrofitted expanded name "Zettabyte File System", it is no longer considered an initialism. —Wikipedia
      - bringtheaction 9y ago
        I don't believe Wikipedia is right. Search site:oracle.com for zettabyte on google and observe the results being documents and blogs from a multitude of years. ZFS being an acronym for Zettabyte File System was neither retrofitted nor all that brief in use it seems. Former employees of Sun Microsystems, I know some of you browse HN. Care to shed some light on this?
        Rapzid 9y ago
        Indeed, LLNL have a humongo Lustre/ZFS cluster and kickstarted the ZFSoL project(mucho gracias to LLNL and Brian Behlendorf) and they refer to it as a Zettabyte File System.
        bcantrill 9y ago
        I wasn't in the room when it happened, but I was probably among the first dozen or so to hear the name -- and it was presented to me as both at once: "ZFS" was both "the last word in filesystems" and (conveniently) a zettabyte. I remember Bonwick quickly pointing out that yes, a yottabyte was larger but "YFS" was passed over for obvious reasons.
        avar 9y ago
        What's that obvious reason?
jchw 9y ago
I don't think Moore's Law held up for much since then, but it would've been hard to know for sure. That being said, ZFS filesystems can span multiple disks, since it does volume management of its own. I don't know if that means we're going to see zettabyte volumes any time soon, though. After all, 2^64 bytes wouldn't be enough to overflow: it would need to be 2^64 blocks...
bhouston 9y ago
I think the largest file systems in the world are Amazon's S3, and maybe Glacier. I guess Google's internal system used for YouTube and Gmail probably outrank it but there are no numbers for it. But I doubt that they are in a single address space on a single file system. So basically we moved away from singular large unified file systems and built swarms of little file systems.
- vesinisa 9y ago
  Indeed. "Cloud" is essentially tons of commodity hardware nodes running a tailored userspace distributed file system application. There now certainly exist cloud services that host well in excess of 2^64 bytes, but they are not hosted on traditional file systems, but an array of those. I can't certainly blame their choice though, and 16-million terabyte hard drive arrays (the limit of 64-bit addressing) on a large mainframe are really just a few quantum leaps away. Heck, I still vividly recall buying a MASSIVE 40 GB HDD in 2000 - it seemed excessive beyond wildest dreams at the time (and was promptly filled to brim at a LAN party.) Since, advances in mass storage tech, most importantly the successful commercialization of perpendicular recording in 2005, mean that nowadays 12 TB drives are commercially viable - roughly 1000x the size of a 40 GB drive. In the next 20 years, petabyte-sized drives might well be available (though almost certainly not based on any form of magnetic recording). Once you hook hundred 50-petabyte drives to a single fileserver, you start closing in on limitations of 64-bit addressing.
  - killjoywashere 9y ago
    Isn't there an XKCD about the limit of flash storage being 3 PB per gallon?
    - mewse 9y ago
      I think you may be referring to an image from this What-If article, where the capacity of a gallon of MicroSD cards is estimated at 1.6PB per gallon: https://what-if.xkcd.com/31/ https://what-if.xkcd.com/31/
runako 9y ago
Backblaze posts some good stuff on HD pricing and reliability. Here's a good chart from their data: https://www.backblaze.com/blog/hard-drive-cost-per-gigabyte/ https://www.backblaze.com/blog/hard-drive-cost-per-gigabyte/ They also link to this great (logarithmic) chart: http://www.mkomo.com/cost-per-gigabyte-update http://www.mkomo.com/cost-per-gigabyte-update TL;DR: price hasn't been dropping by half annually, but prices have come down from ~$0.70/GB to < $0.03/GB now. So roughly a factor of 20 over 13-14 years, somewhere in the neighborhood of 25% improvements compounded annually over that time.
ChuckMcM 9y ago
> So exactly 14 years passed, does someone have 2^64 bytes for a single ZFS filesystem (or anything close to that)? I don't really feel like storage capacity (or 1/price) doubles every year. Its unlikely. There are few opportunities to compile a data set of that size. One might be the NSA collections center in Utah, one might be Google or Microsoft's web cache, and one might be something like the Internet Archive's cache. However the length choice gets 'weird' if you want something larger than 64 bits and less than 128. Your next intermediate choice is 96 bits. That is 64 + 32 (or one additional 32 bit long word. Growing by just 8 means you now have a 9 byte pointer, by 16 gives you a 10 byte pointer. Some architectures are penalized when doing off word alignment, and so on those architectures the structure is padded out to the next word size anyway. I've always found Jef's reasoning in that post amusing but I continue to thing 64 bits would have been fine (just like I think it would have been fine for the Internet v6 work :-). Fortunately time has given us a crap ton of memory for our boxes so the storage penalty isn't too great.
- mjevans 9y ago
  Since 64 bits can serve /many/ use cases and also matches modern machine words I could see there being either two drivers (one with 64 bit native words and the other with full sized ones) or even a fancier driver that uses different ASM bits to just leave half the word all 0s and ignored. The on-disk format could even remain the same; it wouldn't really matter except for lots of small files and those are already so horrid that ZFS might have some other way of handling them. (Like treating /all/ small file names and data as part of a larger directory file or something.)
Bud 9y ago
Moore's Law actually predicted a doubling every two years (for CPUs), not every year, and even the very optimistic Kryder's Law (often cited as the version of Moore's Law for storage) only claimed an increase of 40% per year. Actual progress has fallen far short of that. The rate for 2010-2014 was only 15%/year. Rates were quicker back around 2000-2005, but that pace has fallen dramatically. https://en.wikipedia.org/wiki/Mark_Kryder#Kryder's_law_projection https://en.wikipedia.org/wiki/Mark_Kryder#Kryder's_law_proje...
- wickawic 9y ago
  I think it is reasonable to expect storage to jump dramatically at such a time that we begin storing “3D” data. Not that everything in VR will be voxels or anything, but even if the fractal dimension of our future data is 2.5 it would obliterate our current needs for storage. /speculation
  - AstralStorm 9y ago
    Physics provides us with certain density of information in a volume and that is considering advanced quantum storage. May I remind you that flash, DRAM and SRAM already are 3D structures.
    - Sean1708 9y ago
      I think they meant VR graphics data, not data stored on 3D things.
- mixmastamyk 9y ago
  Wonder what the pace of ssd capacity is?
- gowld 9y ago
  40%/yr is 100%/2years.
madaxe_again 9y ago
CERN are at >100PB, which is only about 6.6 doublings, which is about right with a doubling every two, not one, years. http://iopscience.iop.org/article/10.1088/1742-6596/664/4/042035/pdf http://iopscience.iop.org/article/10.1088/1742-6596/664/4/04...
- sliken 9y ago
  Right, but does CERN connect all 100PB to a single machine?
  - AstralStorm 9y ago
    To a cluster fs which ZFS is not. (And not all of it.)
  - tempay 9y ago
    No. Data is mostly accessed using XRootD[1] which can be used to data from an (almost) unlimited number of nodes. So far as I know, it's optimised for large files so the number of available inodes is limited by RAM. There is also EOS[2] which is based on XRootD and is planned to replace AFS[3] as a general purpose networked filesystem. [1] http://xrootd.org/ http://xrootd.org/ [2] http://eos.web.cern.ch/ http://eos.web.cern.ch/ [3] http://information-technology.web.cern.ch/services/afs-service http://information-technology.web.cern.ch/services/afs-servi...
mrb 9y ago
They had to be on the conservative side, so they assumed a doubling every 9-12 months. In reality we have seen one doubling every ~34 months in the last 14 years. In 2005 I bought 250GB HDDs for $110 each. In 2018 you can find 8TB HDDs for $150 each. That's 5 doublings. So with 5 doublings, if we saw datasets on the order of 2^50 back then, we should see datasets on the order of 2^55 today (30 PB). And sure enough, here is a computer with a 30 PB global filesystem: https://www.fujitsu.com/downloads/TC/sc11/k-computer-system-overview-sc11.pdf https://www.fujitsu.com/downloads/TC/sc11/k-computer-system-... So another 9 doublings to go to 2^64 bytes. With one every 34 months, this should happen in 25 years (2043). But again, if you are a filesystem developer you should be on the conservative side and assume it might double every ~12 months.
- bastawhiz 9y ago
  It's also worth noting that file system developers should think about their file system fifty to a hundred years out. Flash drives today often still come preformatted with FAT32, twenty two years after its release in 1996. Up until a couple years ago, my dad had an external drive formatted with FAT16--a thirty year old file system. If ZFS lives as long as the FATs have (three decades) even at a doubling every 34 months, that means six remaining doublings before 2035. Not quite 2^64, but that's on the liberal side. Half that (doubling every 17 months) is twelve doublings, which is easily over 2^64.
  - ianai 9y ago
    And let’s not forget all the crazy (better word?) features like independence from indianness. They built the file system with an eye toward versatility.
    - mnw21cam 9y ago
      > indianness That has to be one of the worst typos I have ever seen.
      - arghwhat 9y ago
ohazi 9y ago
It was probably incorrect for the original author to cite Moore's law. Moore's law only ever really held for transistor density on integrated circuits, not magnetic storage. Magnetic storage capacity always grew more slowly. Flash storage capacity today has more to do with die stacking than transistor density. In any case, Moore's Law is definitely dead now. Like, for real this time.
Johnny555 9y ago
Backblaze says they are using 400 PB of storage[1], or around 2^59 bytes. Of course, it's not one big filesystem. They aren't quite doubling annually, but looks like they are growing around ~1.5X each year, so in another decade, they could be close to the 2^64 limit. [1] https://www.backblaze.com/company/about.html https://www.backblaze.com/company/about.html
- sliken 9y ago
  Right, but their building block is 48 disks x whatever is best profitable (a mix of 4,6,8,10, and 12TB currently or similar). So each system is 500TB or less, zfs is a filesystem for a single node not a distributed filesystem. Even if we ignore that factor of 800 or so (400PB spread across 500 nodes), they still would need another factor of 128,000 before they would need ZFS's 65th bit. Keep in mind it's 2^128 blocks, not 2^128 bytes. Backblaze would need to consume the entire planets supply of disks for a very long time AND put all those disks connected to a single linux, freebsd, or solaris box. Only then would they need the 65th bit that ZFS has for addressing blocks.
  - ryacko 9y ago
    Not really. Global hard drive production is more than 2^64 bytes or 18 exabytes (SI). https://www.anandtech.com/show/10098/market-views-2015-hard-drive-shipments https://www.anandtech.com/show/10098/market-views-2015-hard-...
    - tempay 9y ago
      From that it looks like production is 130 exabytes for Seagate and Western Digital combined. Using the default block size on ZFS the limit is 128 * 1024 * 2^64 = 2 zettabytes. Even allowing for other manufacturers it's an order of magnitude more than global annual production.
pasbesoin 9y ago
An apparently overlooked aspect of the SpaceX Falcon Heavy launch, is the inclusion of a small quartz disc whose format is expected to scale to something like 260 TB (360?, I forget). And that has a lifespan claimed to potentially extend into the billion plus year range. I don't know about that exact disc/format, but if we and as we are finally able to stably write at such capacities... (whatever technologies end up being used to do so). We're going to run into another "640KB should be enough for anyone", unless we are forward thinking with regard to potential storage format capacities. (And if you think no one will ever be able to consume that much data and detail, just think of all the modeling that will be done and the data behind that. Think of the data being put out by CERN and not just how it will expand but also how people will want to explore lower significance hints, corner cases, and who knows what. Etc.)
anardil 9y ago
A fully decked out Isilon cluster supports 68PB on a single file system. https://www.emc.com/collateral/hardware/white-papers/h10719-isilon-onefs-technical-overview-wp.pdf https://www.emc.com/collateral/hardware/white-papers/h10719-...
wjkohnen 9y ago
ZFS is not really a 128 bit filesystem, it is merely a 64 bit filesystem that is "128 bit ready". I.e. the on disk format is 128 bit, the kernel drivers are not. At least, they were not until Oracle bought Sun, that's about the last time I really cared about ZFS.

CognitiveLens 9y ago

> If 64 bits isn't enough, the next logical step is 128 bits. Can someone explain this? Is there some kind of awkwardness/waste with anything less than doubling the number of bits?

Analemma_ 9y ago
Having an integer/memory address size that isn't a power of two is possible, and has been done before, but it's awkward. Lacking a specific reason not to go to 128 bits, they did that.
- sliken 9y ago
  You mean like doubling the overhead of every pointer into the filesystem? Seems directly related to the rather high ram overhead (I've heard 1GB ram per TB of disk).
  - lmm 9y ago
    Pointers in the filesystem would want to be word-aligned anyway, so once you make them larger than 8 bytes you might as well make them 16 bytes.
    - sliken 9y ago
      Right, but the entire point is that 64 bits is PLENTY. The default block size is 128KB. So 2^64 * block size = 2417851639229258349412352 bytes Or 2147483648 PB. Sure there might be distributed systems that are mind boggling large, but those aren't in a single filesystem attached to a single node. ZFS would be a better filesystem with "only" 64 bit blocks. Today's higher density nodes are something like 48 disks * 12TB, so approximately 2 per PB. 12 per rack would be 6PB. So 178,956,970 racks consuming 300 times the annual production on earth and you'd start wishing for that 65th bit. All in a single OS connecting to a single storage system. Or you could assume that in the next few decades you'd be batshit crazy to want to install that much storage on a single zfs system and half the overhead of every pointer into the filesystem.

znpy 9y ago

"we don't want to deal with this shit for another forty years or possibly forever" would have sufficed for me.

DaiPlusPlus 9y ago
I’m struggling to think of a project that released a new version to accommodate some far-off limit that wasn’t blindsided and obsoleted by something operating on a different paradigm - like using YYMMDD for dates, the changing it to YYYYMMDD for Y2K when they should have been using integer time_t - but then that was 32-bit and now 64-bit. But we won’t even be 10% of the way there before a relativistic-capable time system will replace it. So why worry? We’re doomed to rewrite it anyway :)
- regularfry 9y ago
  Trading off a tiny design decision like that against knowing you're unlikely to have to ever revisit except in an unforeseeable situation is a pretty good place to be.

frozenport 9y ago

Why can't they have a 64 bit and 128 bit ZFS offering side-by-side. When somebody actually needs 128bit they can migrate?

chickenthief 9y ago

"Thus, fully populating a 128-bit storage pool would, literally, require more energy than boiling the oceans.". Could we instead boil some Jupiter Moons?

saalweachter 9y ago
Boiling the oceans is an alternative energy use, not a source.

paulsutter 9y ago

Real-world large filesystems are distributed across many thousands of hosts and multiple datacenters, not mounted as a Linux filesystem on a single host. Because whole racks and whole datacenters fail, not just disk drives. So they used 128 bit because of bikeshedding. Committees always make the most conservative decision possible. Like IPv6.

zzzcpan 9y ago
On the other hand such committee bikeshedding seems to work rather well for PR. It makes them ignore hard problems and instead focus on things most people can understand and relate. Gaining more trust as opposed to a well designed thing with nothing to understand or relate.
kstrauser 9y ago
Except IPv6 being 128-bit makes fast hardware implementations much easier than if it had to deal with shorter prefix lengths. Nothing shorter than 128 really makes sense at all in an IPv4 replacement.
- mixmastamyk 9y ago
  No comprendo. Why would 128 be faster than 64?
  - sliken 9y ago
    IPV6 is basically 64 bits for routing and 64 bits for the local network segment. Seems plausible that it's faster than trying to mask out the bits you need.
    - mixmastamyk 9y ago
      Why not 32:32? Shouldn’t four billion internets be enough?
      - kstrauser 9y ago

ktpsns 9y ago

> Thus, fully populating a 128-bit storage pool would, literally, require more energy than boiling the oceans. This is actually an argument against 128 bit, because it clearly shows that 128 bit are unreachable and thus a waste. What about 96 bit?

DaiPlusPlus 9y ago
96-bit, and other non-power-of-two systems are cumbersome and error-prone to work with in C - which is often used when writing firmware for computer hardware. The real risks of economic damage caused by bit-fiddling bugs is much greater than the risk of bringing the universe’s thermodynamic heat-death ever so slightly nearer...
- speleding 9y ago
  I would think it's fairly easy to keep all calculations in C in 128-bit and just mask out the top few bytes when retrieving and storing. You could also argue that >64 bit values will be rare enough that they warrant their own code path if they are encountered as an optimization (perhaps they already do this?).

russellbeattie 9y ago

Back in 2004, they were probably thinking, "Well, IPv6 is using 128bits, and that's going to be the standard any day now..."

NamTaf 9y ago

What's the downside to having 128 bits over 64? I'm not familiar enough with file systems to know, so I'm left wondering if there's any downside to oversizing the hammer for the nail?

kev009 9y ago
It expands the data structure size, which multiplies N for both on disk and in RAM representation (and cache and shorn cache lines). Many modern architectures have native 128b types (amd64, POWER) so it's not really a big deal to the CPU itself even if you need to do atomic operations for concurrency. OTOH it guarantees there wont really need to be on disk changes for pointer sizes. That may be useful in situations that weren't really imagined, for instance 128b is enough for a Single Level Store where persistent and working set (or NV) are all in the same mappable space for any conceivable workload.
- deleted 9y ago
  [deleted]
dmytrish 9y ago
tl;dr: doubling size of metadata (and usually a half of a 128bit address is zeros anyway). Having twice as much space occupied has many implications: twice as much space lost for data itself, less metadata will fit CPU caches, less throughput in buses between CPU and storage, lengthier read time for the storage itself, etc.

PeCaN 9y ago

> Thus, fully populating a 128-bit storage pool would, literally, require more energy than boiling the oceans. So we should be good until OceanCoin is introduced.

exabrial 9y ago

Is it true zfs never saw widespread adoption because of licensing compat with Linux and patents or has come and gone?

boomboomsubban 9y ago
People are reluctant to use openZFS on Linux as it can't be included in the kernel due to license issues. I'd still say it has widespread adoption when you consider the people willing to run it on Linux and how popular it is on FreeBSD/some other systems.
kikoreis 9y ago
FWIW it has been included in stock Ubuntu since 16.04 LTS so it is widely available, and likely more in use than assumed.
rayvd 9y ago
I think that certainly hurt it. Also, when Oracle bought Sun, they started restricting Solaris use to Sun hardware only. Previously, we were happily using Solaris 10_x86 + ZFS on SuperMicro boxes. We still have north of 7PB of ZFS-based storage at $DAYJOB (NexentaStor based and Dell hardware) after introducing it nearly eight years ago. ZFS itself has been awesome.

natehouk 9y ago

https://www.sec.gov/whistleblower/ https://www.sec.gov/whistleblower/

z3t4 9y ago

It could run the file system of the death star (Star Wars).

sova 9y ago

>That's enough to survive Moore's Law until I'm dead, and after that, it's not my problem. ...really?

mort96 9y ago
No, I don't think that was entirely serious. 128 bytes should be more than enough to last for a very long time, not just a little bit longer than the author's life.
- mort96 9y ago
  Damn, just noticed I wrote 128 bytes and not 128 bits, too late to edit.
  - sova 9y ago
    I wasn't talking about the number of bits, my friend. I was talking about the attitude that "it's not my problem after I'm dead." That's exactly why we are stuck with so many societal problems. For example, the Federal Reserve Act of 1913 was a true victory for the sons-of-guns who got it enacted, but then they died and now we're all left with the aftermath. It's this shortsighted attitude that is the problem, not the number of bits in the implementation.