5 ms·

How many more file systems are we going to see before we realize that the file system is the wrong layer of abstraction to be solving the deduplication problem?

LeoPanthera 10y ago
I think the filesystem is a fine place to do it, I just wish more filesystems would give us the option to do deduplication offline. ZFS's dedup uses grotesque amounts of RAM since it insists on holding the dedup table in it and only doing dedup in realtime. btrfs however allows you to scan the filesystem and do dedup of existing data later (even while it is mounted), using no RAM at all. This is one of a couple of places where I think btrfs is actually becoming a better-designed filesystem than ZFS.
- Skunkleton 10y ago
  There are patches to BTRFS (actively developed) for inband dedup as well.
  - jjobi 10y ago
    BTRFS has so many drawbacks using it as storage driver for docker. See the bottom of the page https://docs.docker.com/engine/userguide/storagedriver/btrfs-driver/ https://docs.docker.com/engine/userguide/storagedriver/btrfs.... Primary drawback being each container creates unique copy of the data in kernel page cache and taking away system resources from applications.
- jjobi 10y ago
  Again, the focus of this blog is not about a de-duplication story, but about a new docker storage driver.
  - LeoPanthera 10y ago
    Well thankfully open discussion is a feature of Hacker News, not a bug.
wmf 10y ago
Solving it at the block layer (device mapper) is even slower, so what do you propose?
- davexunit 10y ago
  Not using image layers to begin with. You can't reasonably do that with Docker or other software that isn't aware of the dependency graphs of the software that it runs. Docker just runs some arbitrary commands and maintains an opaque set of disk images. Seems to be the best you can do there. However, if one were to use software that actually understood the full dependency graph, you wouldn't need disk images at all and you'd be able to share storage without involving special file systems. In GNU Guix, a functional package manager, we can make containers for arbitary software without any duplication because we know the full dependency graph of the application. Applications may share sub-graphs so they are only on disk once. If two programs depend on libpng, libpng and its dependencies are only on disk in one place. With a functional package management system in place, containers become a relatively simple matter of creating namespaces and bind mounting the dependencies. No image layering, no special file systems, just simple bind mounts.
  - wmf 10y ago
    Yeah, Nix is cool but ain't nobody got time to learn it since Dockerfiles are so easy.
    - davexunit 10y ago
      Easy on the surface, but very limiting and frustrating in practice. I think all the energy spent around designing image layer file system after image layer file system is good evidence that the Docker approach isn't as easy as it may seem.
      - gourao 10y ago
        Interesting thought, however I think that the existing graph driver file systems are not actually an "image layer file system"... instead they are a general purpose file systems that are being used to store images. For example, what one wants with launching multiple instances of an image is to be able to clone the image. Today, to accomplish that, file system snapshots are used to simulate this feature. And that is where a lot of the problems start happening. LCFS does things purpose built for these use cases.
old-gregg 10y ago
I am the OP and I kept the original title, but it does not look like a "file system" to me, it looks more like a key/value store optimized for storing container image layers. Also curious to know if anyone is using it in production and what their experiences are.
- jjobi 10y ago
  This is not a general purpose file system, but something purpose built for serving as a docker storage driver. LCFS outperforms existing solutions and does not have any correctness issues. It is an experimental release at this point. We would like to receive feedback from the community and make it successful for everybody. Our tests so far look very promising.
- erichan 10y ago
  Just to add to what jjobi says and agree with point here, we have to battle test this, and the best way to get there is to start in the open. So this is the start :). The fundamental thing here is that (a) it's taking a specific approach [with layers], (b) we're in it for the long-haul, and (c) we want this to be in the open. We’ll update as the battle-testing, production progress continues. So 100% agree with the question/point.
overcast 10y ago
I figure this will probably be shit on here, but have you ever played with Windows Server Deduplication? We use it all over our corporate Windows environment, with a distributed file system, and it averages 48% globally. It doesn't chew up RAM, integrates with shadow copy services, behaves itself while it runs in the background, and I honestly don't see much if any performance hit reading/writing. Now we're not talking petabytes of data, but each of probably a hundred file servers worldwide has 10-20 terabytes worth stored. It's literally set it, and forget it. While you gain 50% of your storage back.
- jjobi 10y ago
  This is a custom built solution for docker storage driver. A de-duplication server cannot be docker storage driver, as there are more things involved in a docker storage driver like layer management, rapid cloning and deletion of those, efficient diffing between layers and stuff like that. The existing solutions are AUFS, Overlay, devicemapper, btrfs and zfs, and a comparison should be made against those.
jjobi 10y ago
We did not solve a typical de-duplication problem with LCFS. If you write two copies of the same data to LCFS, there will be two copies of data stored. We may provide such de-duplication feature at some point in future. Docker is already doing de-duplication by keeping shared data in image layers and expect the storage driver (graphdriver) to manage shared data between many container application efficiently without instantiating multiple copies of shared data in memory and disk. That problem can be solved by well known file system snapshot and cloning technologies and that is what LCFS is doing, and a lot more efficiently than any existing storage drivers out there.
jvinod 10y ago
The focus here is not deduplication. However, if you want to think of it in those terms, it is sharing data at the 'layer' granularity (not block, not file, but a bunch of files) and that's co-ordinated by Docker. The main property of this filesystem is that it is layer aware; it clones layers efficiently and optimizes common workflows in the docker ecosystem.

LeoPanthera 10y ago

I think the filesystem is a fine place to do it, I just wish more filesystems would give us the option to do deduplication offline. ZFS's dedup uses grotesque amounts of RAM since it insists on holding the dedup table in it and only doing dedup in realtime. btrfs however allows you to scan the filesystem and do dedup of existing data later (even while it is mounted), using no RAM at all. This is one of a couple of places where I think btrfs is actually becoming a better-designed filesystem than ZFS.

Skunkleton 10y ago
There are patches to BTRFS (actively developed) for inband dedup as well.
- jjobi 10y ago
  BTRFS has so many drawbacks using it as storage driver for docker. See the bottom of the page https://docs.docker.com/engine/userguide/storagedriver/btrfs-driver/ https://docs.docker.com/engine/userguide/storagedriver/btrfs.... Primary drawback being each container creates unique copy of the data in kernel page cache and taking away system resources from applications.
jjobi 10y ago
Again, the focus of this blog is not about a de-duplication story, but about a new docker storage driver.
- LeoPanthera 10y ago
  Well thankfully open discussion is a feature of Hacker News, not a bug.

wmf 10y ago

Solving it at the block layer (device mapper) is even slower, so what do you propose?

davexunit 10y ago
Not using image layers to begin with. You can't reasonably do that with Docker or other software that isn't aware of the dependency graphs of the software that it runs. Docker just runs some arbitrary commands and maintains an opaque set of disk images. Seems to be the best you can do there. However, if one were to use software that actually understood the full dependency graph, you wouldn't need disk images at all and you'd be able to share storage without involving special file systems. In GNU Guix, a functional package manager, we can make containers for arbitary software without any duplication because we know the full dependency graph of the application. Applications may share sub-graphs so they are only on disk once. If two programs depend on libpng, libpng and its dependencies are only on disk in one place. With a functional package management system in place, containers become a relatively simple matter of creating namespaces and bind mounting the dependencies. No image layering, no special file systems, just simple bind mounts.
- wmf 10y ago
  Yeah, Nix is cool but ain't nobody got time to learn it since Dockerfiles are so easy.
  - davexunit 10y ago
    Easy on the surface, but very limiting and frustrating in practice. I think all the energy spent around designing image layer file system after image layer file system is good evidence that the Docker approach isn't as easy as it may seem.
    - gourao 10y ago
      Interesting thought, however I think that the existing graph driver file systems are not actually an "image layer file system"... instead they are a general purpose file systems that are being used to store images. For example, what one wants with launching multiple instances of an image is to be able to clone the image. Today, to accomplish that, file system snapshots are used to simulate this feature. And that is where a lot of the problems start happening. LCFS does things purpose built for these use cases.

old-gregg 10y ago

I am the OP and I kept the original title, but it does not look like a "file system" to me, it looks more like a key/value store optimized for storing container image layers. Also curious to know if anyone is using it in production and what their experiences are.

jjobi 10y ago
This is not a general purpose file system, but something purpose built for serving as a docker storage driver. LCFS outperforms existing solutions and does not have any correctness issues. It is an experimental release at this point. We would like to receive feedback from the community and make it successful for everybody. Our tests so far look very promising.
erichan 10y ago
Just to add to what jjobi says and agree with point here, we have to battle test this, and the best way to get there is to start in the open. So this is the start :). The fundamental thing here is that (a) it's taking a specific approach [with layers], (b) we're in it for the long-haul, and (c) we want this to be in the open. We’ll update as the battle-testing, production progress continues. So 100% agree with the question/point.

overcast 10y ago

I figure this will probably be shit on here, but have you ever played with Windows Server Deduplication? We use it all over our corporate Windows environment, with a distributed file system, and it averages 48% globally. It doesn't chew up RAM, integrates with shadow copy services, behaves itself while it runs in the background, and I honestly don't see much if any performance hit reading/writing. Now we're not talking petabytes of data, but each of probably a hundred file servers worldwide has 10-20 terabytes worth stored. It's literally set it, and forget it. While you gain 50% of your storage back.

jjobi 10y ago
This is a custom built solution for docker storage driver. A de-duplication server cannot be docker storage driver, as there are more things involved in a docker storage driver like layer management, rapid cloning and deletion of those, efficient diffing between layers and stuff like that. The existing solutions are AUFS, Overlay, devicemapper, btrfs and zfs, and a comparison should be made against those.

jjobi 10y ago

We did not solve a typical de-duplication problem with LCFS. If you write two copies of the same data to LCFS, there will be two copies of data stored. We may provide such de-duplication feature at some point in future. Docker is already doing de-duplication by keeping shared data in image layers and expect the storage driver (graphdriver) to manage shared data between many container application efficiently without instantiating multiple copies of shared data in memory and disk. That problem can be solved by well known file system snapshot and cloning technologies and that is what LCFS is doing, and a lot more efficiently than any existing storage drivers out there.

jvinod 10y ago

The focus here is not deduplication. However, if you want to think of it in those terms, it is sharing data at the 'layer' granularity (not block, not file, but a bunch of files) and that's co-ordinated by Docker. The main property of this filesystem is that it is layer aware; it clones layers efficiently and optimizes common workflows in the docker ecosystem.