5 ms·

Just curious, but what use case do you have for such a voluminous quantity of hardlinks?

amelius 9y ago
Basically copying files without actually copying them. For example, when making versioned backups, or when copying a large number of files from production to various development environments. Something I wish there was a better solution for. Perhaps there is a filesystem out there that supports CoW in a way that fits this usecase (XFS perhaps even), but I haven't looked into it. The disadvantage of using hardlinks is that you can't hardlink between users (user is a property of the file, not of the link to the file), and there's always the danger that a write takes place through one of the links. Imho, that should really be solved at the filesystem level using a CoW scheme.
- koenigdavidmj 9y ago
  Does btrfs, with cp --reflink, not meet your use case?
  - amelius 9y ago
    Yes, thanks, I found that, but I'm not sure if I'm ready for btrfs yet. I have to look into it, and see if btrfs is sufficiently stable, etc.
    - caf 9y ago
      reflinks should work on suitably recent XFS as well.
- e12e 9y ago
  > Perhaps there is a filesystem out there that supports CoW in a way that fits this usecase I'd recommend zfs snapshots - or for something a bit different, maybe nilfs2: http://www.linux-mag.com/id/7345/ http://www.linux-mag.com/id/7345/
- josteink 9y ago
  > Basically copying files without actually copying them. For example, when making versioned backups, or when copying a large number of files from production to various development environments. To me it sounds like you want simple snapshots and backups without the redundancy at the storage level. So why don't use a filesystem which supports that natively, like ZFS or Btrfs?
  - mkj 9y ago
    Do they handle 64k snapshots?
    - lloeki 9y ago
      That's one backup every hour for 7+ years. I'm really curious about the use case. Anyway, theoretically[0]: > As you might know btrfs treats subvolumes as filesystems and hence the number of snapshots is indeed limited: namely by the size of files. According to the btrfs wiki the maximum filesize that can be reached is 2^64 byte == 16 EiB But it seems practically you're hitting the mud at ~100 snapshots[1] but be sure to read the reply to that mail as it will depend on the use case and it might turn out to be fine way beyond that. [0]: https://unix.stackexchange.com/questions/140360/practical-limit-on-the-number-of-btrfs-snapshots#147666 https://unix.stackexchange.com/questions/140360/practical-li... [1]: https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg72385.html https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg...
    - KaiserPro 9y ago
      Not sure, I have ZFS as a docker storage backend, its yet to crash* *not the best data point I realise....
- nialv7 9y ago
  For versioned backup you should use reflink rather than hardlink. Hardlink is not CoW. Modify the file all your backup will change too.
Maakuth 9y ago
git clone uses hardlinks to reuse object storage if you clone a local repository. I use it when I need another working copy, that way you save some space. And time as well as you don't need to pull it all from the network. edit: Local in this case means cloning inside one filesystem. Hardlinks cannot span filesystem boundaries of course.
- xorcist 9y ago
  Just out of curiosity, what is the use case for having multiple local clones? Is it to test run pruning or manipulate the reflog? Or is it for developing on git itself?
  - Maakuth 9y ago
    I typically have some development ongoing on one clone and use the other to do merging for releases and so on.
    - xorcist 9y ago
      Why? Isn't the whole idea of using git that changing branch is just as simple/fast as doing "cd"? You're probably not doing your ongoing work in the branch you are merging for release anyway, if you did you'd have solve any conflicts all over again when you switch clone.

Basically copying files without actually copying them. For example, when making versioned backups, or when copying a large number of files from production to various development environments. Something I wish there was a better solution for. Perhaps there is a filesystem out there that supports CoW in a way that fits this usecase (XFS perhaps even), but I haven't looked into it. The disadvantage of using hardlinks is that you can't hardlink between users (user is a property of the file, not of the link to the file), and there's always the danger that a write takes place through one of the links. Imho, that should really be solved at the filesystem level using a CoW scheme.

koenigdavidmj 9y ago
Does btrfs, with cp --reflink, not meet your use case?
- amelius 9y ago
  Yes, thanks, I found that, but I'm not sure if I'm ready for btrfs yet. I have to look into it, and see if btrfs is sufficiently stable, etc.
  - caf 9y ago
    reflinks should work on suitably recent XFS as well.
e12e 9y ago
> Perhaps there is a filesystem out there that supports CoW in a way that fits this usecase I'd recommend zfs snapshots - or for something a bit different, maybe nilfs2: http://www.linux-mag.com/id/7345/ http://www.linux-mag.com/id/7345/
josteink 9y ago
> Basically copying files without actually copying them. For example, when making versioned backups, or when copying a large number of files from production to various development environments. To me it sounds like you want simple snapshots and backups without the redundancy at the storage level. So why don't use a filesystem which supports that natively, like ZFS or Btrfs?
- mkj 9y ago
  Do they handle 64k snapshots?
  - lloeki 9y ago
    That's one backup every hour for 7+ years. I'm really curious about the use case. Anyway, theoretically[0]: > As you might know btrfs treats subvolumes as filesystems and hence the number of snapshots is indeed limited: namely by the size of files. According to the btrfs wiki the maximum filesize that can be reached is 2^64 byte == 16 EiB But it seems practically you're hitting the mud at ~100 snapshots[1] but be sure to read the reply to that mail as it will depend on the use case and it might turn out to be fine way beyond that. [0]: https://unix.stackexchange.com/questions/140360/practical-limit-on-the-number-of-btrfs-snapshots#147666 https://unix.stackexchange.com/questions/140360/practical-li... [1]: https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg72385.html https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg...
  - KaiserPro 9y ago
    Not sure, I have ZFS as a docker storage backend, its yet to crash* *not the best data point I realise....
nialv7 9y ago
For versioned backup you should use reflink rather than hardlink. Hardlink is not CoW. Modify the file all your backup will change too.

Maakuth 9y ago

git clone uses hardlinks to reuse object storage if you clone a local repository. I use it when I need another working copy, that way you save some space. And time as well as you don't need to pull it all from the network. edit: Local in this case means cloning inside one filesystem. Hardlinks cannot span filesystem boundaries of course.

xorcist 9y ago
Just out of curiosity, what is the use case for having multiple local clones? Is it to test run pruning or manipulate the reflog? Or is it for developing on git itself?
- Maakuth 9y ago
  I typically have some development ongoing on one clone and use the other to do merging for releases and so on.
  - xorcist 9y ago
    Why? Isn't the whole idea of using git that changing branch is just as simple/fast as doing "cd"? You're probably not doing your ongoing work in the branch you are merging for release anyway, if you did you'd have solve any conflicts all over again when you switch clone.