3 ms·
This is a double-edged sword. For our (reasonably savvy) users, being able to duplicate and easily modify individual datasets/files is a feature, not a bug: peo
by nippoo 11y ago
This is a double-edged sword. For our (reasonably savvy) users, being able to duplicate and easily modify individual datasets/files is a feature, not a bug: people can symlink the contents of an entire folder but modify a single array and easily run their analysis on this slightly different dataset, for example.
While it's true that you lose atomicity with this, it can both burn you and help you: you can track specific parts of your dataset in revision control, email subsets of it back and forth, combine datasets easily, or even store it across several servers and manage it with symlinks, for example.
Our users are aware of this and it isn't really a problem for our use-case. But if you're worried, there's always the option of having your whole dataset as a ZIP/TAR file (like all of Microsoft Office file formats are - XML files within a .ZIP); tools for modifying folders within ZIP are much more well-established than HDFView, and most modern programming languages provide libraries to read and modify files within archives without unzipping them; you could make your program agnostic to the files being within an archive (high portability, lower performance) or directly within the FS (loss of atomicity, easier/faster to use).