7 ms·

How difficult is it to preserve metadata when moving a file?

48 points by ashitlerferad 10y ago

8_hours_ago 10y ago
This is part 4 of a series of articles about writing a custom "mv" command. Here's the first one: http://yakking.branchable.com/posts/moving-files-1-copying/ http://yakking.branchable.com/posts/moving-files-1-copying/ Here's the whole list: http://yakking.branchable.com/tags/move/ http://yakking.branchable.com/tags/move/
hackuser 10y ago
I've had a hard time finding a low-end backup solution (i.e., for personal use) that retains and restores all metadata. Long ago, I assumed that backup (and move) applications are relatively simple. But it has never seemed to work out that way.
- pdkl95 10y ago
  Traditionally tar(1) was used as it preserves most metadata. Modern versions even include optional support for ACLs and extended atributes (xattr). It used to be common to see command lines and scripts that piped "tar c" into "tar x" directly as a way of copying files while utilizing tar's metadata-preserving features. Today, back-to-back tar is rarely needed, as cp itself already has options to preserve metadata. In particular, "-a","--archive" turns all the normal features (including --recursive), making metadata-preserving backups as easy as: cp -a $SRC $DST which is a shorthand for cp --recursive --no-dereference \ --preserve=context,links,mode,ownership,timestamps,xattr Just the "-a" option is enough for most simple local backups. Fancier backups are possible with cp's "--backup=numbered" and --reflink=auto features, but my favorite just-works solution for backups is rdiff-backup[1]. It preserves everything, uses rsync internally for minimal data transfer on incremental backups, and many more features. [1] http://www.nongnu.org/rdiff-backup/ http://www.nongnu.org/rdiff-backup/
  - jhallenworld 10y ago
    I still use tar, for example to copy an entire Linux distribution to a new drive is just: mount /dev/sdb1 /m tar cf - . --exclude=./m --exclude=./proc --exclude=./sys | (cd /m; umask 0; tar -xf -) mkdir /m/proc /m/sys grub-install --root-directory=/m /dev/sdb (except you must also adjust UUIDs in /etc/fstab)
- jbronn 10y ago
  I'm a big fan of fsarchiver (https://www.fsarchiver.org/ https://www.fsarchiver.org/). I've successfully used it for restoring full Windows 7 systems (NTFS), which implicitly requires getting all the metadata correct (e.g., alternate data streams).
  - hackuser 10y ago
    Thanks; it looks great on first impression. I'll give it a try.
irrational 10y ago
Apparently extremely difficult to impossible since the date created on my photos never seems to stick.
- Terretta 10y ago
  Use 'jhead' and sync file timestamp to exif data: http://www.sentex.net/~mwandel/jhead/ http://www.sentex.net/~mwandel/jhead/
95014_refugee 10y ago
Hopefully at some point you will mention the fact that not everyone has the same idea about what a file is (streams, extended attributes) or what metadata is (ACLs, versions, other timestamps)...
djcapelis 10y ago
A discussion of extended attributes, ACLs and security labels (if you want those too) appear to be missing? These things are a deep dark pit of problems and it isn't entirely unreasonable to just give up on them, but every once in awhile someone tries to do something useful with them so it's at least important to be aware that they exist.
- kimballkinnison 10y ago
  I believe Richard has those planned for another article, this is a series still in development :-)
  - djcapelis 10y ago
    Ahh. :)
todd8 10y ago
Don't forget MS NTFS files support multiple data streams [1] and OS X supports resource forks [2]. I dislike both of these partly because they don't fit within the set of posix commands and command options I'm familiar with. [1] https://technet.microsoft.com/en-us/library/cc961812.aspx https://technet.microsoft.com/en-us/library/cc961812.aspx [2] https://en.wikipedia.org/wiki/Resource_fork https://en.wikipedia.org/wiki/Resource_fork

48 points by ashitlerferad 10y ago

This is part 4 of a series of articles about writing a custom "mv" command. Here's the first one: http://yakking.branchable.com/posts/moving-files-1-copying/ http://yakking.branchable.com/posts/moving-files-1-copying/ Here's the whole list: http://yakking.branchable.com/tags/move/ http://yakking.branchable.com/tags/move/

hackuser 10y ago

I've had a hard time finding a low-end backup solution (i.e., for personal use) that retains and restores all metadata. Long ago, I assumed that backup (and move) applications are relatively simple. But it has never seemed to work out that way.

pdkl95 10y ago
Traditionally tar(1) was used as it preserves most metadata. Modern versions even include optional support for ACLs and extended atributes (xattr). It used to be common to see command lines and scripts that piped "tar c" into "tar x" directly as a way of copying files while utilizing tar's metadata-preserving features. Today, back-to-back tar is rarely needed, as cp itself already has options to preserve metadata. In particular, "-a","--archive" turns all the normal features (including --recursive), making metadata-preserving backups as easy as: cp -a $SRC $DST which is a shorthand for cp --recursive --no-dereference \ --preserve=context,links,mode,ownership,timestamps,xattr Just the "-a" option is enough for most simple local backups. Fancier backups are possible with cp's "--backup=numbered" and --reflink=auto features, but my favorite just-works solution for backups is rdiff-backup[1]. It preserves everything, uses rsync internally for minimal data transfer on incremental backups, and many more features. [1] http://www.nongnu.org/rdiff-backup/ http://www.nongnu.org/rdiff-backup/
- jhallenworld 10y ago
  I still use tar, for example to copy an entire Linux distribution to a new drive is just: mount /dev/sdb1 /m tar cf - . --exclude=./m --exclude=./proc --exclude=./sys | (cd /m; umask 0; tar -xf -) mkdir /m/proc /m/sys grub-install --root-directory=/m /dev/sdb (except you must also adjust UUIDs in /etc/fstab)
jbronn 10y ago
I'm a big fan of fsarchiver (https://www.fsarchiver.org/ https://www.fsarchiver.org/). I've successfully used it for restoring full Windows 7 systems (NTFS), which implicitly requires getting all the metadata correct (e.g., alternate data streams).
- hackuser 10y ago
  Thanks; it looks great on first impression. I'll give it a try.

irrational 10y ago

Apparently extremely difficult to impossible since the date created on my photos never seems to stick.

Terretta 10y ago
Use 'jhead' and sync file timestamp to exif data: http://www.sentex.net/~mwandel/jhead/ http://www.sentex.net/~mwandel/jhead/

95014_refugee 10y ago

Hopefully at some point you will mention the fact that not everyone has the same idea about what a file is (streams, extended attributes) or what metadata is (ACLs, versions, other timestamps)...

djcapelis 10y ago

A discussion of extended attributes, ACLs and security labels (if you want those too) appear to be missing? These things are a deep dark pit of problems and it isn't entirely unreasonable to just give up on them, but every once in awhile someone tries to do something useful with them so it's at least important to be aware that they exist.

kimballkinnison 10y ago
I believe Richard has those planned for another article, this is a series still in development :-)
- djcapelis 10y ago
  Ahh. :)

todd8 10y ago

Don't forget MS NTFS files support multiple data streams [1] and OS X supports resource forks [2]. I dislike both of these partly because they don't fit within the set of posix commands and command options I'm familiar with. [1] https://technet.microsoft.com/en-us/library/cc961812.aspx https://technet.microsoft.com/en-us/library/cc961812.aspx [2] https://en.wikipedia.org/wiki/Resource_fork https://en.wikipedia.org/wiki/Resource_fork