3 ms·

The problem -- and I've been burned on both sides of this -- is that you need either a container file, or you need users to understand that a directory is essen

by x0x0 11y ago

The problem -- and I've been burned on both sides of this -- is that you need either a container file, or you need users to understand that a directory is essentially a file. Not only does this complicate users lives when they want to move what they, quite reasonably, view as a single file between different computers or back it up, but they can and will remove individual pieces. Or copy it around and have some of the pieces not show up and be very confused that copy/move operations -- particularly to/from a NAS -- are now nothing like atomic. Another thing that will happen is this: if you just use a directory full of files as a single logical file, you will end up writing code that does the equivalent of 'rm -rf ${somedir}' because when users choose to overwrite "files" (really, a directory), you need to clear out any previous data so experiment runs don't get mixed. You can easily see where this can go bad; you will have to take extraordinary care.

rdtsc 11y ago
Depending on the setup, moves / exporting / snapshots can be helped by fancy-shmancy file systems that allow snapshots (well LVM does too but I hear it is slow). But generally means BTRFS or ZFS. The pattern is to create a read-only snapshot of directory. Then feed that snapshot to tar (maybe apply some mild compression on it depending on your tradeoffs). Then delete snapshot. Even better directly use BTRFS (or ZFS) send / recv to send to move to another system with that file system. Can also pipe tar directly to something like nc. Remember benchmarked moving a large number of files on a LAN and tar + nc came out as the winner over say rsync or scp. It saturated 1Gbps connection pretty close to its maximum expected capacity. You'd obviously not expect user to do that by hand and would write tools to do it.
nippoo 11y ago
This is a double-edged sword. For our (reasonably savvy) users, being able to duplicate and easily modify individual datasets/files is a feature, not a bug: people can symlink the contents of an entire folder but modify a single array and easily run their analysis on this slightly different dataset, for example. While it's true that you lose atomicity with this, it can both burn you and help you: you can track specific parts of your dataset in revision control, email subsets of it back and forth, combine datasets easily, or even store it across several servers and manage it with symlinks, for example. Our users are aware of this and it isn't really a problem for our use-case. But if you're worried, there's always the option of having your whole dataset as a ZIP/TAR file (like all of Microsoft Office file formats are - XML files within a .ZIP); tools for modifying folders within ZIP are much more well-established than HDFView, and most modern programming languages provide libraries to read and modify files within archives without unzipping them; you could make your program agnostic to the files being within an archive (high portability, lower performance) or directly within the FS (loss of atomicity, easier/faster to use).

by x0x0 11y ago

rdtsc 11y ago

Depending on the setup, moves / exporting / snapshots can be helped by fancy-shmancy file systems that allow snapshots (well LVM does too but I hear it is slow). But generally means BTRFS or ZFS. The pattern is to create a read-only snapshot of directory. Then feed that snapshot to tar (maybe apply some mild compression on it depending on your tradeoffs). Then delete snapshot. Even better directly use BTRFS (or ZFS) send / recv to send to move to another system with that file system. Can also pipe tar directly to something like nc. Remember benchmarked moving a large number of files on a LAN and tar + nc came out as the winner over say rsync or scp. It saturated 1Gbps connection pretty close to its maximum expected capacity. You'd obviously not expect user to do that by hand and would write tools to do it.

nippoo 11y ago

This is a double-edged sword. For our (reasonably savvy) users, being able to duplicate and easily modify individual datasets/files is a feature, not a bug: people can symlink the contents of an entire folder but modify a single array and easily run their analysis on this slightly different dataset, for example. While it's true that you lose atomicity with this, it can both burn you and help you: you can track specific parts of your dataset in revision control, email subsets of it back and forth, combine datasets easily, or even store it across several servers and manage it with symlinks, for example. Our users are aware of this and it isn't really a problem for our use-case. But if you're worried, there's always the option of having your whole dataset as a ZIP/TAR file (like all of Microsoft Office file formats are - XML files within a .ZIP); tools for modifying folders within ZIP are much more well-established than HDFView, and most modern programming languages provide libraries to read and modify files within archives without unzipping them; you could make your program agnostic to the files being within an archive (high portability, lower performance) or directly within the FS (loss of atomicity, easier/faster to use).