7 ms·

LCFS: A New Container Filesystem for Modern Datacenters

71 points by old-gregg 10y ago

gourao 10y ago
Hi, I am one of the contributors on this project, and happy to answer any questions... LCFS will also be supporting the snapshot driver discussed here - https://github.com/docker/containerd/pull/484 https://github.com/docker/containerd/pull/484. Currently, the install experience is not the greatest... We rely on the Docker v2 plugin interface and that is still in beta. We are working with the Docker team to help us smooth out some of these rough edges. LCFS will also support other container formats and any help from the community is much appreciated.
- akanet 10y ago
  This looks amazing. One of the biggest pains of using docker is that the storage driver inevitably explodes the system. Is anyone using LCFS in production? Why did you get started building it?
  - gourao 10y ago
    LCFS is not yet being used in production as far as I know We started building this because there are a lot of issues running containers in production, specifically around image space and inode management and also with page cash usage. This file system was specifically designed to address memory and storage management of container images
CSDude 10y ago
Really nice to see the new file systems addressing this issue, but, as many including me hurt by AUFS, I am being cautious. Do you think it is battle tested enough? As it is different than other merging ones, it is a little more dangerous than them as well since it works on block level? Not trying to be negative, I really liked it, and I am not even happy about even a docker load command performance over my overlay2, and will defintely be trying LCFS.
- jjobi 10y ago
  It is a mini-file system, working directly on top of a block device. Having something else in between is not a good idea. Much of the data handled by docker storage driver is temporary and does not even have to hit disk before thrown away. This project is currently in experimental mode. We would appreciate any feedback you may have.
davexunit 10y ago
How many more file systems are we going to see before we realize that the file system is the wrong layer of abstraction to be solving the deduplication problem?
- LeoPanthera 10y ago
  I think the filesystem is a fine place to do it, I just wish more filesystems would give us the option to do deduplication offline. ZFS's dedup uses grotesque amounts of RAM since it insists on holding the dedup table in it and only doing dedup in realtime. btrfs however allows you to scan the filesystem and do dedup of existing data later (even while it is mounted), using no RAM at all. This is one of a couple of places where I think btrfs is actually becoming a better-designed filesystem than ZFS.
  - Skunkleton 10y ago
    There are patches to BTRFS (actively developed) for inband dedup as well.
    - jjobi 10y ago
      BTRFS has so many drawbacks using it as storage driver for docker. See the bottom of the page https://docs.docker.com/engine/userguide/storagedriver/btrfs-driver/ https://docs.docker.com/engine/userguide/storagedriver/btrfs.... Primary drawback being each container creates unique copy of the data in kernel page cache and taking away system resources from applications.
  - jjobi 10y ago
    Again, the focus of this blog is not about a de-duplication story, but about a new docker storage driver.
    - LeoPanthera 10y ago
      Well thankfully open discussion is a feature of Hacker News, not a bug.
softwarelimits 10y ago
Does anybody know of a good deduplication script / tool that runs on every filesystem and can be used as a nightly cronjob?
- kruhft 10y ago
  Here's a quick one if your filesystem supports hard links: #!/bin/ksh typeset -A hashes find . -type f -exec shasum -a 512 {} \; | while read hash filename; do if [[ -z ${hashes[$hash]} ]]; then hashes[$hash]="$filename" else echo ln -f "'${hashes[$hash]}'" "'$filename'" fi done Pipe to /bin/sh if you're happy with the results.
  - rsanders 10y ago
    Good answer, but unless you mark the files read-only/immutable, this ought to have big red flags all over it. Userland just can't do this safely in the general case.
    - kruhft 10y ago
      Good point. This is for data that's not in use.
  - khc 10y ago
    Do you actually use this? Seems at the minimum you want to use -xdev with find and do something better than ln -f.
    - kruhft 10y ago
      Yes, I forgot to put in -xdev, but what's wrong with ln -f?
- nabla9 10y ago
  fdupes -rL /dir see also jdupes fork of fdupes https://github.com/jbruchon/jdupes https://github.com/jbruchon/jdupes https://github.com/adrianlopezroche/fdupes https://github.com/adrianlopezroche/fdupes
jpgvm 10y ago
It's fuse based however, so that has some caveats. Interesting project though, could see a native port one day maybe.
- deleted 10y ago
  [deleted]
- gourao 10y ago
  Fuse was used to avoid Kernel dependency and also to be cross platform compatible
- jjobi 10y ago
  This was done using fuse to make it portable across different distros. We have not seen any major issues testing it as a docker storage driver, compared to the graphdrivers in use today.

71 points by old-gregg 10y ago

Hi, I am one of the contributors on this project, and happy to answer any questions... LCFS will also be supporting the snapshot driver discussed here - https://github.com/docker/containerd/pull/484 https://github.com/docker/containerd/pull/484. Currently, the install experience is not the greatest... We rely on the Docker v2 plugin interface and that is still in beta. We are working with the Docker team to help us smooth out some of these rough edges. LCFS will also support other container formats and any help from the community is much appreciated.

akanet 10y ago
This looks amazing. One of the biggest pains of using docker is that the storage driver inevitably explodes the system. Is anyone using LCFS in production? Why did you get started building it?
- gourao 10y ago
  LCFS is not yet being used in production as far as I know We started building this because there are a lot of issues running containers in production, specifically around image space and inode management and also with page cash usage. This file system was specifically designed to address memory and storage management of container images

CSDude 10y ago

Really nice to see the new file systems addressing this issue, but, as many including me hurt by AUFS, I am being cautious. Do you think it is battle tested enough? As it is different than other merging ones, it is a little more dangerous than them as well since it works on block level? Not trying to be negative, I really liked it, and I am not even happy about even a docker load command performance over my overlay2, and will defintely be trying LCFS.

jjobi 10y ago
It is a mini-file system, working directly on top of a block device. Having something else in between is not a good idea. Much of the data handled by docker storage driver is temporary and does not even have to hit disk before thrown away. This project is currently in experimental mode. We would appreciate any feedback you may have.

davexunit 10y ago

How many more file systems are we going to see before we realize that the file system is the wrong layer of abstraction to be solving the deduplication problem?

LeoPanthera 10y ago
I think the filesystem is a fine place to do it, I just wish more filesystems would give us the option to do deduplication offline. ZFS's dedup uses grotesque amounts of RAM since it insists on holding the dedup table in it and only doing dedup in realtime. btrfs however allows you to scan the filesystem and do dedup of existing data later (even while it is mounted), using no RAM at all. This is one of a couple of places where I think btrfs is actually becoming a better-designed filesystem than ZFS.
- Skunkleton 10y ago
  There are patches to BTRFS (actively developed) for inband dedup as well.
  - jjobi 10y ago
    BTRFS has so many drawbacks using it as storage driver for docker. See the bottom of the page https://docs.docker.com/engine/userguide/storagedriver/btrfs-driver/ https://docs.docker.com/engine/userguide/storagedriver/btrfs.... Primary drawback being each container creates unique copy of the data in kernel page cache and taking away system resources from applications.
- jjobi 10y ago
  Again, the focus of this blog is not about a de-duplication story, but about a new docker storage driver.
  - LeoPanthera 10y ago
    Well thankfully open discussion is a feature of Hacker News, not a bug.

softwarelimits 10y ago

Does anybody know of a good deduplication script / tool that runs on every filesystem and can be used as a nightly cronjob?

kruhft 10y ago
Here's a quick one if your filesystem supports hard links: #!/bin/ksh typeset -A hashes find . -type f -exec shasum -a 512 {} \; | while read hash filename; do if [[ -z ${hashes[$hash]} ]]; then hashes[$hash]="$filename" else echo ln -f "'${hashes[$hash]}'" "'$filename'" fi done Pipe to /bin/sh if you're happy with the results.
- rsanders 10y ago
  Good answer, but unless you mark the files read-only/immutable, this ought to have big red flags all over it. Userland just can't do this safely in the general case.
  - kruhft 10y ago
    Good point. This is for data that's not in use.
- khc 10y ago
  Do you actually use this? Seems at the minimum you want to use -xdev with find and do something better than ln -f.
  - kruhft 10y ago
    Yes, I forgot to put in -xdev, but what's wrong with ln -f?
nabla9 10y ago
fdupes -rL /dir see also jdupes fork of fdupes https://github.com/jbruchon/jdupes https://github.com/jbruchon/jdupes https://github.com/adrianlopezroche/fdupes https://github.com/adrianlopezroche/fdupes

jpgvm 10y ago

It's fuse based however, so that has some caveats. Interesting project though, could see a native port one day maybe.

deleted 10y ago
[deleted]
gourao 10y ago
Fuse was used to avoid Kernel dependency and also to be cross platform compatible
jjobi 10y ago
This was done using fuse to make it portable across different distros. We have not seen any major issues testing it as a docker storage driver, compared to the graphdrivers in use today.