4 ms·

What makes having a lot of files in a directory a hard problem?

justinclift 11y ago
From memory, it was something about not having a central metadata server combined with needing to talk with each of the nodes to get their idea of "the latest version" of file info. Especially tricky when taking into account edge cases, such as when some nodes are failed/down/etc. Unfortunately, I don't remember most of the details. It's fallen out of my headspace in the months since I've left. ;)
TallGuyShort 11y ago
It's not uncommon with distributed storage to err one way or another, because you have to store large amounts of data and metadata. Designing a system that does both extremely well is non-trivial. You can optimize to store a few very large files or many very small files, but doing both while scaling reliably is harder than you might think.
- DannoHung 11y ago
  Naively it seems weird since metadata is just data, right? So if you can store a large amount of data in a file, you should be able to store the metadata But I don't know shit about filesystem design, soooooo...
  - TallGuyShort 11y ago
    Well the metadata I'm referring to is the metadata required to locate a file: mappings from directory structure, filename, possibly permissions / ownership to the blocks and replicas distributed across a cluster. If you get to the point where you need start locating metadata because it might also be arbtirarily distributed you now need meta-metadata. And I'm sure you can see how complex that's going to get :)
notacoward 11y ago
Hi. I'm one of the GlusterFS developers. Hi, Justin. ;) The basic answer to your question is that networks are slow compared to local storage. In order to get decent performance, you must either avoid network round trips or amortize their cost over many operations. We - not just GlusterFS but distributed filesystems in general - can do this pretty well for some operations. We can batch, buffer, cache, etc. This works great for plain old reads and writes to large files (for example). It doesn't work so well for operations that have to touch many small files. For those, in order to ensure the required level of consistency/currency, we have two choices. (1) Send a request per file to get current metadata. (2) Cache metadata, and participate in some sort of consistency protocol to make sure we don't serve stale cached information. Both approaches have workloads where they perform better and workloads where they perform worse. In addition, the second approach adds a lot of complexity, especially in a system where failures are common and a loosely coordinated set of servers must respond (systems with a single master server have an easier time here but are less resilient). The inherent difficulty of this approach is why e.g. CephFS has taken so long to mature. GlusterFS took the first route instead. It does mean that "ZOT" (Zillions Of Tiny files) workloads will perform poorly. I won't deny that. On the other hand, it's easier to test or prove correct, and the time not spent on solving the hard version of that problem - often for little eventual benefit - can instead be spent on other kinds of improvements. Some people are happy with that. Some are not. Some spread FUD. Some try to implement the practical equivalent of a distributed filesystem on top of some alternative (e.g. object stores) with their own even more serious limitations, and experience even more pain as a result. Some are initially unhappy with these tradeoffs, but work with us and learn to work more effectively with these limitations to enjoy other benefits. That's life in the big city.
- justinclift 11y ago
  Thanks for clearing that up, that's awesome. :)

From memory, it was something about not having a central metadata server combined with needing to talk with each of the nodes to get their idea of "the latest version" of file info. Especially tricky when taking into account edge cases, such as when some nodes are failed/down/etc. Unfortunately, I don't remember most of the details. It's fallen out of my headspace in the months since I've left. ;)

TallGuyShort 11y ago

It's not uncommon with distributed storage to err one way or another, because you have to store large amounts of data and metadata. Designing a system that does both extremely well is non-trivial. You can optimize to store a few very large files or many very small files, but doing both while scaling reliably is harder than you might think.

DannoHung 11y ago
Naively it seems weird since metadata is just data, right? So if you can store a large amount of data in a file, you should be able to store the metadata But I don't know shit about filesystem design, soooooo...
- TallGuyShort 11y ago
  Well the metadata I'm referring to is the metadata required to locate a file: mappings from directory structure, filename, possibly permissions / ownership to the blocks and replicas distributed across a cluster. If you get to the point where you need start locating metadata because it might also be arbtirarily distributed you now need meta-metadata. And I'm sure you can see how complex that's going to get :)

notacoward 11y ago

Hi. I'm one of the GlusterFS developers. Hi, Justin. ;) The basic answer to your question is that networks are slow compared to local storage. In order to get decent performance, you must either avoid network round trips or amortize their cost over many operations. We - not just GlusterFS but distributed filesystems in general - can do this pretty well for some operations. We can batch, buffer, cache, etc. This works great for plain old reads and writes to large files (for example). It doesn't work so well for operations that have to touch many small files. For those, in order to ensure the required level of consistency/currency, we have two choices. (1) Send a request per file to get current metadata. (2) Cache metadata, and participate in some sort of consistency protocol to make sure we don't serve stale cached information. Both approaches have workloads where they perform better and workloads where they perform worse. In addition, the second approach adds a lot of complexity, especially in a system where failures are common and a loosely coordinated set of servers must respond (systems with a single master server have an easier time here but are less resilient). The inherent difficulty of this approach is why e.g. CephFS has taken so long to mature. GlusterFS took the first route instead. It does mean that "ZOT" (Zillions Of Tiny files) workloads will perform poorly. I won't deny that. On the other hand, it's easier to test or prove correct, and the time not spent on solving the hard version of that problem - often for little eventual benefit - can instead be spent on other kinds of improvements. Some people are happy with that. Some are not. Some spread FUD. Some try to implement the practical equivalent of a distributed filesystem on top of some alternative (e.g. object stores) with their own even more serious limitations, and experience even more pain as a result. Some are initially unhappy with these tradeoffs, but work with us and learn to work more effectively with these limitations to enjoy other benefits. That's life in the big city.

justinclift 11y ago
Thanks for clearing that up, that's awesome. :)