4 ms·

They had to be on the conservative side, so they assumed a doubling every 9-12 months. In reality we have seen one doubling every ~34 months in the last 14 year

by mrb 9y ago

They had to be on the conservative side, so they assumed a doubling every 9-12 months. In reality we have seen one doubling every ~34 months in the last 14 years. In 2005 I bought 250GB HDDs for $110 each. In 2018 you can find 8TB HDDs for $150 each. That's 5 doublings. So with 5 doublings, if we saw datasets on the order of 2^50 back then, we should see datasets on the order of 2^55 today (30 PB). And sure enough, here is a computer with a 30 PB global filesystem: https://www.fujitsu.com/downloads/TC/sc11/k-computer-system-overview-sc11.pdf https://www.fujitsu.com/downloads/TC/sc11/k-computer-system-... So another 9 doublings to go to 2^64 bytes. With one every 34 months, this should happen in 25 years (2043). But again, if you are a filesystem developer you should be on the conservative side and assume it might double every ~12 months.

bastawhiz 9y ago
It's also worth noting that file system developers should think about their file system fifty to a hundred years out. Flash drives today often still come preformatted with FAT32, twenty two years after its release in 1996. Up until a couple years ago, my dad had an external drive formatted with FAT16--a thirty year old file system. If ZFS lives as long as the FATs have (three decades) even at a doubling every 34 months, that means six remaining doublings before 2035. Not quite 2^64, but that's on the liberal side. Half that (doubling every 17 months) is twelve doublings, which is easily over 2^64.
- ianai 9y ago
  And let’s not forget all the crazy (better word?) features like independence from indianness. They built the file system with an eye toward versatility.
  - mnw21cam 9y ago
    > indianness That has to be one of the worst typos I have ever seen.
    - arghwhat 9y ago
      The filesystem does not share any culture with Native America and India.
      - MatthewWilkes 9y ago
        The term does derive from Gullivers Travels, where the big endians and the little endians were both to be found on an island in the Indian ocean. Of course, the real reason for the mistake is that phone spelling correction will convert endian to Indian.
        ianai 9y ago
        It felt wrong at the time but it’s been too long for me to remember the correct spelling.
- Moter8 9y ago
  >If 64 bits isn't enough, the next logical step is 128 bits. >That's enough to survive Moore's Law until I'm dead, and after that, it's not my problem ;)
- AlphaSite 9y ago
  HFS stuck around for around 30 years as well. And is still shipped on some systems.
gonzo 9y ago
In 2016, Backblaze swapped out more than 3,500 2 TB HGST and WDC hard drives for 2,400 8 TB Seagate drives. That’s 7 PB to 19.2 PB. If they had replaced all 3,500 drives it would be 4X.
- mbreese 9y ago
  But that's not all running under a single filesystem. It's multiple file systems that are orchestrated together to form their network storage space. I think that's the point from some of the other comments (below). Instead of having every larger and larger single filesystems, once you get to a certain point (~1-2PB is my guess), it simply isn't worth it to organize your data under a single filesystem. That's not to say ZFS was wrong in choosing 128 bits... at the time ZFS was designed, the level of horizontal partitioning that we are now accustomed to just wasn't a thing. Compared to HDD times, networks were far too slow to support such a thing. Now, however, we talk about HDDs being too slow compared to our networks. At the time, 128 bits was a solid choice, even if it was overkill. Plus, Sun was definitely a scale up kind of company... if you want to get bigger, by a bigger ($$$$) box. So having a large capacity FS was good for their bottom line. Now, we think in terms of scaling out with multiple (cheaper) smaller boxes... which is probably the only way we've managed with current levels of data.
  - arghwhat 9y ago
    > It's multiple file systems that are orchestrated together to form their network storage space. No, that is not true. A vault acts as a single unit, effectively making it a single filesystem as much as a ZFS pool is a single filesystem. Internally, a vault operates on a set of "tomes", which are groups of 1 harddrive from each storage pod (20). These "tomes" handle the redundancy aspect as well. However, the "tome" is not the filesystem. The concepts map cleanly to the ZFS world: A vault is equivalent to a ZFS pool, and a tome is equivalent to a ZFS vdev in redundant configuration. Neither vdevs nor tomes are filesystems on their own—they only become a filesystem when combined into pools and vaults, respectively. Now, your argument does hold if interpreted differently: One does not use a normal filesystem for those absurd sizes. However, it is a single filesystem.
jerven 9y ago
Just hit df -lh on the EBI login nodes and see a 22PB filesystem at 93% full. I suspect if I dig deeper I will find a bigger one as they had 120PB of storage last year[1]. [1]https://academic.oup.com/nar/article/46/D1/D21/4658830 https://academic.oup.com/nar/article/46/D1/D21/4658830

by mrb 9y ago

bastawhiz 9y ago

It's also worth noting that file system developers should think about their file system fifty to a hundred years out. Flash drives today often still come preformatted with FAT32, twenty two years after its release in 1996. Up until a couple years ago, my dad had an external drive formatted with FAT16--a thirty year old file system. If ZFS lives as long as the FATs have (three decades) even at a doubling every 34 months, that means six remaining doublings before 2035. Not quite 2^64, but that's on the liberal side. Half that (doubling every 17 months) is twelve doublings, which is easily over 2^64.

ianai 9y ago
And let’s not forget all the crazy (better word?) features like independence from indianness. They built the file system with an eye toward versatility.
- mnw21cam 9y ago
  > indianness That has to be one of the worst typos I have ever seen.
  - arghwhat 9y ago
    The filesystem does not share any culture with Native America and India.
    - MatthewWilkes 9y ago
      The term does derive from Gullivers Travels, where the big endians and the little endians were both to be found on an island in the Indian ocean. Of course, the real reason for the mistake is that phone spelling correction will convert endian to Indian.
      - ianai 9y ago
        It felt wrong at the time but it’s been too long for me to remember the correct spelling.
Moter8 9y ago
>If 64 bits isn't enough, the next logical step is 128 bits. >That's enough to survive Moore's Law until I'm dead, and after that, it's not my problem ;)
AlphaSite 9y ago
HFS stuck around for around 30 years as well. And is still shipped on some systems.

gonzo 9y ago

In 2016, Backblaze swapped out more than 3,500 2 TB HGST and WDC hard drives for 2,400 8 TB Seagate drives. That’s 7 PB to 19.2 PB. If they had replaced all 3,500 drives it would be 4X.

mbreese 9y ago
But that's not all running under a single filesystem. It's multiple file systems that are orchestrated together to form their network storage space. I think that's the point from some of the other comments (below). Instead of having every larger and larger single filesystems, once you get to a certain point (~1-2PB is my guess), it simply isn't worth it to organize your data under a single filesystem. That's not to say ZFS was wrong in choosing 128 bits... at the time ZFS was designed, the level of horizontal partitioning that we are now accustomed to just wasn't a thing. Compared to HDD times, networks were far too slow to support such a thing. Now, however, we talk about HDDs being too slow compared to our networks. At the time, 128 bits was a solid choice, even if it was overkill. Plus, Sun was definitely a scale up kind of company... if you want to get bigger, by a bigger ($$$$) box. So having a large capacity FS was good for their bottom line. Now, we think in terms of scaling out with multiple (cheaper) smaller boxes... which is probably the only way we've managed with current levels of data.
- arghwhat 9y ago
  > It's multiple file systems that are orchestrated together to form their network storage space. No, that is not true. A vault acts as a single unit, effectively making it a single filesystem as much as a ZFS pool is a single filesystem. Internally, a vault operates on a set of "tomes", which are groups of 1 harddrive from each storage pod (20). These "tomes" handle the redundancy aspect as well. However, the "tome" is not the filesystem. The concepts map cleanly to the ZFS world: A vault is equivalent to a ZFS pool, and a tome is equivalent to a ZFS vdev in redundant configuration. Neither vdevs nor tomes are filesystems on their own—they only become a filesystem when combined into pools and vaults, respectively. Now, your argument does hold if interpreted differently: One does not use a normal filesystem for those absurd sizes. However, it is a single filesystem.

jerven 9y ago

Just hit df -lh on the EBI login nodes and see a 22PB filesystem at 93% full. I suspect if I dig deeper I will find a bigger one as they had 120PB of storage last year[1]. [1]https://academic.oup.com/nar/article/46/D1/D21/4658830 https://academic.oup.com/nar/article/46/D1/D21/4658830