4 ms·

tar cf metadata.tar.zstd *.json in a large directory of image files with json sidecars Somebody will say use a database, but when working for example with ML

tar cf metadata.tar.zstd *.json in a large directory of image files with json sidecars Somebody will say use a database, but when working for example with ML training data one label file per image is a common setup and what most tooling expects, and this extends further up the data preparation chain

NegativeK 1y ago
I won't say use a database, but I will beg you to compress a parent directory instead of making a tar bomb.
- mort96 1y ago
  That really just makes the problem worse: tar czf whatever.tgz whatever/*.json; you've added 9 bytes to every file path. I mean I get that you're suggesting to provide only one directory on the argv. But it sucks that the above solution to add json files to an archive while ignoring non-json files only works below some not-insane number of files.
  - stouset 1y ago
    Having an insane number of files in the same directory is already a performance killer. Here you’re talking about having something like 10,000 JSON files in one directory plus some number of non-JSON files, and you’d just be better off in all cases having these things split across separate directories. Does it suck you can’t use globing for this situation? Sure, yeah, fine, but by the time it’s a problem you’re already starting to push the limits of other parts of the system too. Also using that glob is definitely going to bite you when you forget some day that some of the files you needed were in subdirectories.
    - wongarsu 1y ago
      But in this example the files are conceptually a flat structure. Any hierarchy would be artificial, like the common ./[first two letters]/[second two letters]/filename structure. Which you can do, but it certainly doesn't make creating the above tarball any easier. Now we really need to use some kind of `find` invocation instead of a simple glob It also just extends the original question. If I have a system with 96GB RAM and terabytes of fast SSD storage, why shouldn't I be able to put tens of thousands of files in a directory and write a glob that matches half of them? I get that this was inconceivable in v6 unix, but in modern times those are entirely reasonable numbers. Heck, Windows Explorer can do that in a GUI, on a network drive. And that's a program that has been treated as essentially feature complete for nearly 30 years now, on an OS with a famously slow file system stack. Why shouldn't I be able to do the same on a linux command line?
    - mort96 1y ago
      > Does it suck you can’t use globing for this situation? Sure, yeah Then we agree :)
cesarb 1y ago
> tar cf metadata.tar.zstd *.json From a quick look at the tar manual, there is a --files-from option to read more command line parameters from a file; I haven't tried, but you could probably combine it with find through bash's process substitution to create the list of files on the fly.
- syncsynchalt 1y ago
  Yes, workarounds exist, but it would be nicer if the arbitrary limits were removed instead.
badc0ffee 1y ago
You don't need to tar everything in one command. You can batch your tar into multiple commands with a reasonable amount of arguments with something like `rm metadata.tar.zstd && find . -maxdepth 1 -name \*.json -exec tar rf metadata.tar.zstd {} +`.

NegativeK 1y ago

I won't say use a database, but I will beg you to compress a parent directory instead of making a tar bomb.

mort96 1y ago
That really just makes the problem worse: tar czf whatever.tgz whatever/*.json; you've added 9 bytes to every file path. I mean I get that you're suggesting to provide only one directory on the argv. But it sucks that the above solution to add json files to an archive while ignoring non-json files only works below some not-insane number of files.
- stouset 1y ago
  Having an insane number of files in the same directory is already a performance killer. Here you’re talking about having something like 10,000 JSON files in one directory plus some number of non-JSON files, and you’d just be better off in all cases having these things split across separate directories. Does it suck you can’t use globing for this situation? Sure, yeah, fine, but by the time it’s a problem you’re already starting to push the limits of other parts of the system too. Also using that glob is definitely going to bite you when you forget some day that some of the files you needed were in subdirectories.
  - wongarsu 1y ago
    But in this example the files are conceptually a flat structure. Any hierarchy would be artificial, like the common ./[first two letters]/[second two letters]/filename structure. Which you can do, but it certainly doesn't make creating the above tarball any easier. Now we really need to use some kind of `find` invocation instead of a simple glob It also just extends the original question. If I have a system with 96GB RAM and terabytes of fast SSD storage, why shouldn't I be able to put tens of thousands of files in a directory and write a glob that matches half of them? I get that this was inconceivable in v6 unix, but in modern times those are entirely reasonable numbers. Heck, Windows Explorer can do that in a GUI, on a network drive. And that's a program that has been treated as essentially feature complete for nearly 30 years now, on an OS with a famously slow file system stack. Why shouldn't I be able to do the same on a linux command line?
  - mort96 1y ago
    > Does it suck you can’t use globing for this situation? Sure, yeah Then we agree :)

cesarb 1y ago

> tar cf metadata.tar.zstd *.json From a quick look at the tar manual, there is a --files-from option to read more command line parameters from a file; I haven't tried, but you could probably combine it with find through bash's process substitution to create the list of files on the fly.

syncsynchalt 1y ago
Yes, workarounds exist, but it would be nicer if the arbitrary limits were removed instead.

badc0ffee 1y ago

You don't need to tar everything in one command. You can batch your tar into multiple commands with a reasonable amount of arguments with something like `rm metadata.tar.zstd && find . -maxdepth 1 -name \*.json -exec tar rf metadata.tar.zstd {} +`.