4 ms·

Cool project and kudos for open sourcing it. Noteworthy limitation: > TernFS should not be used for tiny files — our median file size is 2MB.

heipei 1y ago
Yeah, that was the first thing I checked as well. Being suited for small / tiny files is a great property of the SeaweedFS system.
pandemic_region 1y ago
What happens if you put a tiny file on it then? Bad perf, possible file corruption, ... ?
- redundantly 1y ago
  Probably wasting space and lower performance.
- jleahy 1y ago
  It's just not optimised for tiny files. It absolutely would work with no problems at all, and you could definitely use it to store 100 billion 1kB files with zero problems (and that is 100 terabytes of data, probably on flash, so no joke). However you can't use it to store 1 exabyte of 1 kilobyte files (at least not yet).
- KaiserPro 1y ago
  Bad space efficiency and possibly exhausting your inode system (ie there is space left on the device, but you can't put any files on it)
jandrewrogers 1y ago
I have worked on exabyte-scale storage engines. There is a good engineering reason for this type of limitation. If you had 1 KiB average file size then you have quadrillions of metadata objects to quickly search and manage with fine-granularity. The kinds of operations and coordination you need to do with metadata is difficult to achieve reliably when the metadata structure itself is many PB in size. There are interesting edge cases that show up when you have to do deep paging of this metadata off of storage. Making this not slow requires unorthodox and unusual design choices that introduce a lot of complexity. Almost none of the metadata fits in memory, including many parts of conventional architectures we assume will always fit in memory. A mere trillion objects is right around the limit of where the allocators, metadata, etc can be made to scale with heroic efforts before conventional architectures break down and things start to become deeply weird on the software design side. Storage engines need to be reliable, so avoiding that design frontier makes a lot of sense if you can avoid it. It is possible to break this barrier but it introduces myriad interesting design and computer science problems for which there is little literature.
- stuartjohnson12 1y ago
  This sounds like a fascinating niche piece of technical expertise I would love to hear more about. What are the biggest challenges in scaling metadata from a trillion to a quadrillion objects?
  - jandrewrogers 1y ago
    It is dependent on the intended workload but there are a few common design problems. Keep in mind that you can't just deal in the average case, you have to design for the worst possible cases of extremely skewed or pathologically biased distributions. A lot of the design work is proving worst case resource bounds under various scenarios and then proving the worst case behavior of designs intended to mitigate that. An obvious one is bulk deletion, which is rarely fast at any scale. This may involve trillions of updates to search indexing structures, which in naive implementations could look like pointer-chasing across disk. Releasing storage to allocators has no locality because you are streaming the allocations to release off that storage in semi-random order. It is unhelpfully resistant to most scheduling-based locality optimization techniques. You also want to parallelize this as much as possible and some of these allocators will be global-ish. The most interesting challenge to me is meta-scheduling. Cache replacement algorithms usually don't provide I/O locality at this scale so standard mitigations for cache-resistant workloads like dynamic schedule rewriting and latency-hiding are used instead. Schedulers are in the center of the hot path so you really want these to be memory-resident and fast. Their state size is loosely correlated with the number of objects, so in some extreme cases these can easily exceed available memory on large servers. You can address this by designing a "meta-scheduler" that adaptively optimizes the scheduling of scheduler state, so that the right bits are memory-resident at the right time so that the scheduler can optimally schedule its workload. It is difficult to overstate how much of a breaking change to conventional architecture this turns out to be. These add some value even if the state is memory resident but they greatly increase design complexity and make tail latencies more difficult to manage. A more basic challenge is that you start dealing with numbers that may not be representable in 64-bits. Similarly, many popular probabilistic algorithms may not offer what you need when the number of entities is this large. I aggressively skirted these issues for a long time before relenting. I deal more with database storage engines than filesystems, but to a first approximation "files" and "shards" are equivalent for these purposes.
Eikon 1y ago
Shameless plug: https://github.com/Barre/ZeroFS https://github.com/Barre/ZeroFS I initially developed it for a usecase where I needed to store billions of tiny files, and it just requires a single s3 bucket as infrastructure.
- Datagenerator 1y ago
  Interesting, can I use SeaweedFS as bucket provider?
  - Eikon 1y ago
    If SeaweedFS supports conditional PUTs, yes.
    - mdaniel 1y ago
      It claims to have it: https://github.com/seaweedfs/seaweedfs/wiki/S3-Conditional-Operations#conditional-put-optimistic-locking https://github.com/seaweedfs/seaweedfs/wiki/S3-Conditional-O...
- HackerNewt-doms 1y ago
  > ZeroFS - The Filesystem That Makes S3 your Primary Storage. What is the motivation to use s3 as primary storage?
stonogo 1y ago
...which places it firmly in the "just like every other so-called exascale file system." We already had GPFS...

heipei 1y ago

Yeah, that was the first thing I checked as well. Being suited for small / tiny files is a great property of the SeaweedFS system.

pandemic_region 1y ago

What happens if you put a tiny file on it then? Bad perf, possible file corruption, ... ?

redundantly 1y ago
Probably wasting space and lower performance.
jleahy 1y ago
It's just not optimised for tiny files. It absolutely would work with no problems at all, and you could definitely use it to store 100 billion 1kB files with zero problems (and that is 100 terabytes of data, probably on flash, so no joke). However you can't use it to store 1 exabyte of 1 kilobyte files (at least not yet).
KaiserPro 1y ago
Bad space efficiency and possibly exhausting your inode system (ie there is space left on the device, but you can't put any files on it)

jandrewrogers 1y ago

I have worked on exabyte-scale storage engines. There is a good engineering reason for this type of limitation. If you had 1 KiB average file size then you have quadrillions of metadata objects to quickly search and manage with fine-granularity. The kinds of operations and coordination you need to do with metadata is difficult to achieve reliably when the metadata structure itself is many PB in size. There are interesting edge cases that show up when you have to do deep paging of this metadata off of storage. Making this not slow requires unorthodox and unusual design choices that introduce a lot of complexity. Almost none of the metadata fits in memory, including many parts of conventional architectures we assume will always fit in memory. A mere trillion objects is right around the limit of where the allocators, metadata, etc can be made to scale with heroic efforts before conventional architectures break down and things start to become deeply weird on the software design side. Storage engines need to be reliable, so avoiding that design frontier makes a lot of sense if you can avoid it. It is possible to break this barrier but it introduces myriad interesting design and computer science problems for which there is little literature.

stuartjohnson12 1y ago
This sounds like a fascinating niche piece of technical expertise I would love to hear more about. What are the biggest challenges in scaling metadata from a trillion to a quadrillion objects?
- jandrewrogers 1y ago
  It is dependent on the intended workload but there are a few common design problems. Keep in mind that you can't just deal in the average case, you have to design for the worst possible cases of extremely skewed or pathologically biased distributions. A lot of the design work is proving worst case resource bounds under various scenarios and then proving the worst case behavior of designs intended to mitigate that. An obvious one is bulk deletion, which is rarely fast at any scale. This may involve trillions of updates to search indexing structures, which in naive implementations could look like pointer-chasing across disk. Releasing storage to allocators has no locality because you are streaming the allocations to release off that storage in semi-random order. It is unhelpfully resistant to most scheduling-based locality optimization techniques. You also want to parallelize this as much as possible and some of these allocators will be global-ish. The most interesting challenge to me is meta-scheduling. Cache replacement algorithms usually don't provide I/O locality at this scale so standard mitigations for cache-resistant workloads like dynamic schedule rewriting and latency-hiding are used instead. Schedulers are in the center of the hot path so you really want these to be memory-resident and fast. Their state size is loosely correlated with the number of objects, so in some extreme cases these can easily exceed available memory on large servers. You can address this by designing a "meta-scheduler" that adaptively optimizes the scheduling of scheduler state, so that the right bits are memory-resident at the right time so that the scheduler can optimally schedule its workload. It is difficult to overstate how much of a breaking change to conventional architecture this turns out to be. These add some value even if the state is memory resident but they greatly increase design complexity and make tail latencies more difficult to manage. A more basic challenge is that you start dealing with numbers that may not be representable in 64-bits. Similarly, many popular probabilistic algorithms may not offer what you need when the number of entities is this large. I aggressively skirted these issues for a long time before relenting. I deal more with database storage engines than filesystems, but to a first approximation "files" and "shards" are equivalent for these purposes.

Eikon 1y ago

Shameless plug: https://github.com/Barre/ZeroFS https://github.com/Barre/ZeroFS I initially developed it for a usecase where I needed to store billions of tiny files, and it just requires a single s3 bucket as infrastructure.

Datagenerator 1y ago
Interesting, can I use SeaweedFS as bucket provider?
- Eikon 1y ago
  If SeaweedFS supports conditional PUTs, yes.
  - mdaniel 1y ago
    It claims to have it: https://github.com/seaweedfs/seaweedfs/wiki/S3-Conditional-Operations#conditional-put-optimistic-locking https://github.com/seaweedfs/seaweedfs/wiki/S3-Conditional-O...
HackerNewt-doms 1y ago
> ZeroFS - The Filesystem That Makes S3 your Primary Storage. What is the motivation to use s3 as primary storage?

stonogo 1y ago

...which places it firmly in the "just like every other so-called exascale file system." We already had GPFS...