Large datasets can be tricky to handle, as the normal workflows people take for granted may no longer work as expected. For example, many filesystems from nix

Large datasets can be tricky to handle, as the normal workflows people take for granted may no longer work as expected. For example, many filesystems from nix will scale just fine, self-check, and de-duplicate. However, accessing a path in a high-branching factor tree can cause problems for ls or rm etc. Notably, for external BLOBS we found it simple to convert a filename into its sha512 hash with standard formatted media specific extensions, and include the extracted meta-data file in json (details of the encoding, label, and stats etc.) Thus, for quality of life improvements we would pack the sub-paths based on the file hash characters... so the smaller leaf path content entries were present in a given path (char[0] is local index, char[1..k] is the sub paths.) It is strange, but when the files get big it is convenient to be able to audit each part in a decoupled way independent of the underlying filesystems/NFS/databases. the file hash stored inside a database also infers the host OS file set label, and packed location * the metadata is preserved along side the media in human readable utf8 json text, and thus the host node does not require knowledge of the media specific encoding during most operations (i.e. the file-server has minimal dependencies.) * the file external-BLOB location is set by the size of k-1 hash string length used (k=128 chars in hash512 for example) * normal CLI still works in each k-1 leaf path, and will be under ((2+w)16files) entries depending on how its implemented. Note, Windows usually demands k<124 sub-paths deep, and caps string lengths to under 260 chars. BLOB file sets are trivially accessed/locked by many processes on the host OS, and additional files with extensions/analytics may be put beside the original media as json metadata etc. * operates on top of whatever filesystem you are deploying at the moment, where we assume k<=128 will sit under your file system limits * user and group read-only permissions are supported on the host NFS or JBOD * Must turn-off auto-indexing filesystem search routines on the host OS node * a database BLOB index can be rebuilt/ported from the archive tree leaf nodes * source file corruption is self-evident (the metadata should match inside the json file as well) * contents are obfuscated, but duplicate file checks are trivial (note this differs from block level de-duplication, which can be impractical if the data flow rate is high) * tree-root-node sub-paths may be externally mounted on explicitly sharded volumes to share/backup the workload as needed... given the pseudo-random hash makes specific io-busy areas unlikely. Mind you we only had to deal with only around 40TiB of sparse video data on that old project. Unsure if such a method would be performant for 2M hours of content. Sometimes the metadata format inside media is versioned in nonstandard ways... if your intent is long term access support, than storing stats on the program and version used to read the file may be necessary to parse it properly in the future (even if a VM OS image snapshot with the codecs/parser is also archived as a read-only file.) Best of luck =3