5 ms·

> If 64 bits isn't enough, the next logical step is 128 bits. Can someone explain this? Is there some kind of awkwardness/waste with anything less than doublin

> If 64 bits isn't enough, the next logical step is 128 bits. Can someone explain this? Is there some kind of awkwardness/waste with anything less than doubling the number of bits?

Analemma_ 9y ago
Having an integer/memory address size that isn't a power of two is possible, and has been done before, but it's awkward. Lacking a specific reason not to go to 128 bits, they did that.
- sliken 9y ago
  You mean like doubling the overhead of every pointer into the filesystem? Seems directly related to the rather high ram overhead (I've heard 1GB ram per TB of disk).
  - lmm 9y ago
    Pointers in the filesystem would want to be word-aligned anyway, so once you make them larger than 8 bytes you might as well make them 16 bytes.
    - sliken 9y ago
      Right, but the entire point is that 64 bits is PLENTY. The default block size is 128KB. So 2^64 * block size = 2417851639229258349412352 bytes Or 2147483648 PB. Sure there might be distributed systems that are mind boggling large, but those aren't in a single filesystem attached to a single node. ZFS would be a better filesystem with "only" 64 bit blocks. Today's higher density nodes are something like 48 disks * 12TB, so approximately 2 per PB. 12 per rack would be 6PB. So 178,956,970 racks consuming 300 times the annual production on earth and you'd start wishing for that 65th bit. All in a single OS connecting to a single storage system. Or you could assume that in the next few decades you'd be batshit crazy to want to install that much storage on a single zfs system and half the overhead of every pointer into the filesystem.
  - mirashii 9y ago
    That 1GB of RAM per 1TB of disk is a recommendation from the ZFS documentation, but let's remember the audience, and the feature sets enabled when we talk about it. In particular, that suggestion stemmed from running in a configuration with file deduplication enabled and heavy amounts of caching, which ZFS is made to take advantage of. The high memory usage profile definitely isn't from an extra 4 bytes on a pointer, but from design and features of the filesystem.
  - snuxoll 9y ago
    The recommendation of RAM to disk is for storage servers expecting to take advantage of the ZFS ARC and possibly deduplication. We've got something like 192GB in our TrueNAS appliances at work and it's always full of cached data, we don't bother with dedup since it doesn't benefit our workload (ECM image storage) and would bring the system to a halt with tens of millions of 4KB image files.
  - _jal 9y ago
    Others mentioned that those rules of thumb are for running the deduplication system, which one really shouldn't[1]. In real-world use memory needs will vary by use like anything else, but are entirely reasonable. I have an old box with 4 gigs of RAM and about 20T disk that performs just fine for modest file-serving needs. If I had more than a few users for that system, it would need more at some point, but mostly for client access software, not the filesystem. ZFS will use as much memory as you want, and benefits from lots of it in many use cases. But it doesn't require it. I find it to be the current sweet-spot between useful features, performance, and stability - snapshots, trivial filesystem serialization; not the fastest, but acceptable; and rock solid. Anyone approaching it from scratch, I highly recommend thoroughly going through the operations one does without your irreplaceable data on it. Everyone, including me, ignores this recommendation. So at the very least, allow me to suggest that you think very long and hard before running that -f (force) command the first time you replace a disk on the array with the baby pics. [1] Maybe on a system with a lot of ram but constrained storage; I haven't encountered those, but I'm sure they're out there somewhere.
ronsor 9y ago
Memory alignment? It's also the next power of 2. Anything non-power-of-two would be cumbersome.
abpavel 9y ago
Computers are binary. After 2^6 the next step is 2^7. Anything in between would be awkward.
- kzrdude 9y ago
  So the next step after 2^64 would be 2^65 then, or 65-bit.
  - deleted 9y ago
    [deleted]
  - mjevans 9y ago
    The number of bits in the length of the word should be a power of 2; thus the next logical 'full word' after 64 bits is 128 bits. 2 to the n... 0 = 1 1 = 2 2 = 4 3 = 8 4 = 16 5 = 32 6 = 64 7 = 128 For large files it probably doesn't make sense to worry about the size of metadata. For smaller files, particularly ones that haven't changed recently, it might make more sense to use some kind of compression scheme. A decade later, we have a lot of decent compression schemes out of patent and, likely based on their now public methods, a revitalization of development towards schemes that are more optimal for different use cases. Some combination of pre-filtering stages and compression for large caches of small files (like a directory of source code objects or configurations that are infrequently read) might make sense today.
phihag_ 9y ago
Yes, any operation is easy in 2^(2^n). For instance, take addition of two 128-bit numbers x and y (seen as 64-bit int arrays) on a 64-bit big-endian CPU: sum[1] = x[1] + y[1] sum[0] = x[0] + y[0] + carry from previous operation In contrast, if you'd use 96 bits, you couldn't just use 64 bit integer operations. Instead, you'd have to cast a lot: sum[4..11] = *((int64*) x) + *((int64*) y) sum[0..3] = (int32) ( (int64) *((int32*) x) + (int64) *((int32*) y) + carry) So you'd read 32 bit-values into 64 bit registers, set the top 32 bits to zero, perform the addition, and then write out a 32bit value again. It gets much worse if your CPU architecture does not support the addition to 2^(2^n); if you were to use 100 bits, you'd have to AND the values with a bitmask, and write out single bytes. So 128 is far easier to implement, faster on many CPU architectures, plus you get the peace of mind that your code works for a long time. For instance, let's assume the lower bound of 9 months per doubling (which is unrealistic as described in this article), then you're going to hit: 50 bits (baseline from article): 2004 64 bits: 2014 80 bits: 2026 92 bits: 2035 100 bits: 2040 128 bits: 2062 Now, what's the expected lifetime of a long-term storage system? It's well-known that the US nuclear force uses 8 inch floppy disks. Those were designed around 1970. So a lifetime of roughly 50 years is to be expected. For ZFS, that would be 2054. By this (admittedly very conservative) calculation, 128 bits is only barely more than required.
- garmaine 9y ago
  On the other hand they could have used 96bits of block pointer and 32bits of meta data in a sort of tagged reference or capability system, instead of shuffling around a bunch of high order zero bytes forever.
  - Samis2001 9y ago
    Assuming the tagged reference or capability system was built, wouldn't it need software to take advantage of it? If it's not actively used, no real point having it over more block pointer space - and I doubt significant amounts of software would use such a filesystem-specific feature.
  - kstrauser 9y ago
    Is there any advantage in a CPU to doing 32-bit math instead of 128-bit? My first guess is that this would make pointer operations much slower.

> If 64 bits isn't enough, the next logical step is 128 bits. Can someone explain this? Is there some kind of awkwardness/waste with anything less than doubling the number of bits?

Analemma_ 9y ago

Having an integer/memory address size that isn't a power of two is possible, and has been done before, but it's awkward. Lacking a specific reason not to go to 128 bits, they did that.

sliken 9y ago
You mean like doubling the overhead of every pointer into the filesystem? Seems directly related to the rather high ram overhead (I've heard 1GB ram per TB of disk).
- lmm 9y ago
  Pointers in the filesystem would want to be word-aligned anyway, so once you make them larger than 8 bytes you might as well make them 16 bytes.
  - sliken 9y ago
    Right, but the entire point is that 64 bits is PLENTY. The default block size is 128KB. So 2^64 * block size = 2417851639229258349412352 bytes Or 2147483648 PB. Sure there might be distributed systems that are mind boggling large, but those aren't in a single filesystem attached to a single node. ZFS would be a better filesystem with "only" 64 bit blocks. Today's higher density nodes are something like 48 disks * 12TB, so approximately 2 per PB. 12 per rack would be 6PB. So 178,956,970 racks consuming 300 times the annual production on earth and you'd start wishing for that 65th bit. All in a single OS connecting to a single storage system. Or you could assume that in the next few decades you'd be batshit crazy to want to install that much storage on a single zfs system and half the overhead of every pointer into the filesystem.
- mirashii 9y ago
  That 1GB of RAM per 1TB of disk is a recommendation from the ZFS documentation, but let's remember the audience, and the feature sets enabled when we talk about it. In particular, that suggestion stemmed from running in a configuration with file deduplication enabled and heavy amounts of caching, which ZFS is made to take advantage of. The high memory usage profile definitely isn't from an extra 4 bytes on a pointer, but from design and features of the filesystem.
- snuxoll 9y ago
  The recommendation of RAM to disk is for storage servers expecting to take advantage of the ZFS ARC and possibly deduplication. We've got something like 192GB in our TrueNAS appliances at work and it's always full of cached data, we don't bother with dedup since it doesn't benefit our workload (ECM image storage) and would bring the system to a halt with tens of millions of 4KB image files.
- _jal 9y ago
  Others mentioned that those rules of thumb are for running the deduplication system, which one really shouldn't[1]. In real-world use memory needs will vary by use like anything else, but are entirely reasonable. I have an old box with 4 gigs of RAM and about 20T disk that performs just fine for modest file-serving needs. If I had more than a few users for that system, it would need more at some point, but mostly for client access software, not the filesystem. ZFS will use as much memory as you want, and benefits from lots of it in many use cases. But it doesn't require it. I find it to be the current sweet-spot between useful features, performance, and stability - snapshots, trivial filesystem serialization; not the fastest, but acceptable; and rock solid. Anyone approaching it from scratch, I highly recommend thoroughly going through the operations one does without your irreplaceable data on it. Everyone, including me, ignores this recommendation. So at the very least, allow me to suggest that you think very long and hard before running that -f (force) command the first time you replace a disk on the array with the baby pics. [1] Maybe on a system with a lot of ram but constrained storage; I haven't encountered those, but I'm sure they're out there somewhere.

ronsor 9y ago

Memory alignment? It's also the next power of 2. Anything non-power-of-two would be cumbersome.

abpavel 9y ago

Computers are binary. After 2^6 the next step is 2^7. Anything in between would be awkward.

kzrdude 9y ago
So the next step after 2^64 would be 2^65 then, or 65-bit.
- deleted 9y ago
  [deleted]
- mjevans 9y ago
  The number of bits in the length of the word should be a power of 2; thus the next logical 'full word' after 64 bits is 128 bits. 2 to the n... 0 = 1 1 = 2 2 = 4 3 = 8 4 = 16 5 = 32 6 = 64 7 = 128 For large files it probably doesn't make sense to worry about the size of metadata. For smaller files, particularly ones that haven't changed recently, it might make more sense to use some kind of compression scheme. A decade later, we have a lot of decent compression schemes out of patent and, likely based on their now public methods, a revitalization of development towards schemes that are more optimal for different use cases. Some combination of pre-filtering stages and compression for large caches of small files (like a directory of source code objects or configurations that are infrequently read) might make sense today.

phihag_ 9y ago

Yes, any operation is easy in 2^(2^n). For instance, take addition of two 128-bit numbers x and y (seen as 64-bit int arrays) on a 64-bit big-endian CPU: sum[1] = x[1] + y[1] sum[0] = x[0] + y[0] + carry from previous operation In contrast, if you'd use 96 bits, you couldn't just use 64 bit integer operations. Instead, you'd have to cast a lot: sum[4..11] = *((int64*) x) + *((int64*) y) sum[0..3] = (int32) ( (int64) *((int32*) x) + (int64) *((int32*) y) + carry) So you'd read 32 bit-values into 64 bit registers, set the top 32 bits to zero, perform the addition, and then write out a 32bit value again. It gets much worse if your CPU architecture does not support the addition to 2^(2^n); if you were to use 100 bits, you'd have to AND the values with a bitmask, and write out single bytes. So 128 is far easier to implement, faster on many CPU architectures, plus you get the peace of mind that your code works for a long time. For instance, let's assume the lower bound of 9 months per doubling (which is unrealistic as described in this article), then you're going to hit: 50 bits (baseline from article): 2004 64 bits: 2014 80 bits: 2026 92 bits: 2035 100 bits: 2040 128 bits: 2062 Now, what's the expected lifetime of a long-term storage system? It's well-known that the US nuclear force uses 8 inch floppy disks. Those were designed around 1970. So a lifetime of roughly 50 years is to be expected. For ZFS, that would be 2054. By this (admittedly very conservative) calculation, 128 bits is only barely more than required.

garmaine 9y ago
On the other hand they could have used 96bits of block pointer and 32bits of meta data in a sort of tagged reference or capability system, instead of shuffling around a bunch of high order zero bytes forever.
- Samis2001 9y ago
  Assuming the tagged reference or capability system was built, wouldn't it need software to take advantage of it? If it's not actively used, no real point having it over more block pointer space - and I doubt significant amounts of software would use such a filesystem-specific feature.
- kstrauser 9y ago
  Is there any advantage in a CPU to doing 32-bit math instead of 128-bit? My first guess is that this would make pointer operations much slower.