4 ms·

You can recompile your kernel to work around the 100k-ish command line length limit: https://stackoverflow.com/questions/33051108/how-to-get-around-the-linux-to

You can recompile your kernel to work around the 100k-ish command line length limit: https://stackoverflow.com/questions/33051108/how-to-get-around-the-linux-too-many-arguments-limit https://stackoverflow.com/questions/33051108/how-to-get-arou... However, that sounds like solving the wrong end of the problem to me. I don't really know what a 4k JPEG worth of command line arguments is even supposed to be used for.

tomrod 1y ago
A huge security manifold to encourage adoption then sell white hat services on top?
dataflow 1y ago
> I don't really know what a 4k JPEG worth of command line arguments is even supposed to be used for. I didn't either, until I learned about compiler command line flags. But also: the nice thing about conmand line flags is they aren't persisted anywhere (normally). That's good for security.
- eightys3v3n 1y ago
  There were ways for other users on the system to see running process command line flags I thought. This isn't so good for security.
  - dataflow 1y ago
    That depends on your threat model.
- Ghoelian 1y ago
  Pretty sure `.bash_history` includes arguments as well.
  - dataflow 1y ago
    That's only if you're launching from an interactive Bash? We're talking about subprocess launches in general.
    - johnisgood 1y ago
      If you add a whitespace before the command, it will not even get appended to history!
      - amelius 1y ago
        This is quite annoying behavior, actually.
        nosrepa 1y ago
        Good thing you can turn it off!
        amelius 1y ago
        Yeah, took me a while to figure that out though. Plus I lost my history. Typing an extra space should not invoke advanced functionality. Bad design, etc.
      - zamadatix 1y ago
        I had never run across this and it didn't work for me when I tried it. After some reading it looks like the HISTCONTROL variable needs to be set to include "ignorespace" or "ignoreboth" (the other option included in this is "ignoredups"). This would be really killer if it was always enabled and the same across shells but "some shells support something akin and you have to check if it is actually enabled on the ones that do" is just annoying enough that I probably won't bother adopting this on my local machine even though it sounds convenient as a concept.
        wongarsu 1y ago
        I'm mostly using debian and ubuntu flavors (both on desktop and the cloud images provided and customized by various cloud and hosting providers) and they have all had this as the default behavior for bash YMMV with other shells and base distros
wongarsu 1y ago
tar cf metadata.tar.zstd *.json in a large directory of image files with json sidecars Somebody will say use a database, but when working for example with ML training data one label file per image is a common setup and what most tooling expects, and this extends further up the data preparation chain
- NegativeK 1y ago
  I won't say use a database, but I will beg you to compress a parent directory instead of making a tar bomb.
  - mort96 1y ago
    That really just makes the problem worse: tar czf whatever.tgz whatever/*.json; you've added 9 bytes to every file path. I mean I get that you're suggesting to provide only one directory on the argv. But it sucks that the above solution to add json files to an archive while ignoring non-json files only works below some not-insane number of files.
    - stouset 1y ago
      Having an insane number of files in the same directory is already a performance killer. Here you’re talking about having something like 10,000 JSON files in one directory plus some number of non-JSON files, and you’d just be better off in all cases having these things split across separate directories. Does it suck you can’t use globing for this situation? Sure, yeah, fine, but by the time it’s a problem you’re already starting to push the limits of other parts of the system too. Also using that glob is definitely going to bite you when you forget some day that some of the files you needed were in subdirectories.
      - wongarsu 1y ago
        But in this example the files are conceptually a flat structure. Any hierarchy would be artificial, like the common ./[first two letters]/[second two letters]/filename structure. Which you can do, but it certainly doesn't make creating the above tarball any easier. Now we really need to use some kind of `find` invocation instead of a simple glob It also just extends the original question. If I have a system with 96GB RAM and terabytes of fast SSD storage, why shouldn't I be able to put tens of thousands of files in a directory and write a glob that matches half of them? I get that this was inconceivable in v6 unix, but in modern times those are entirely reasonable numbers. Heck, Windows Explorer can do that in a GUI, on a network drive. And that's a program that has been treated as essentially feature complete for nearly 30 years now, on an OS with a famously slow file system stack. Why shouldn't I be able to do the same on a linux command line?
silverwind 1y ago
There's numerous examples of useful long commands, here is one: perl -p -i -e 's#foo#bar#g' **/*
- PaulDavisThe1st 1y ago
  no limits: find . -type f -exec perl -p -i -e 's#foo#bar#g' {} \;
  - Someone 1y ago
    That runs perl multiple times, possibly/likely often in calls that effectively are no-ops. To optimize the number of invocations of perl, you can/should use xargs (with -0)
    - dr4g0n 1y ago
      No need for `xargs` in this case, `find` has been able to take care of this for quite some time now, using `+` instead of `;`: find . -type f -exec perl -p -i -e 's#foo#bar#g' {} +
    - tremon 1y ago
      xargs constructs a command line from the find results, so if **/* exceeds the max command line length, so will xargs.
      - Someone 1y ago
        xargs was written to avoid that problem, so no, it won’t. https://man7.org/linux/man-pages/man1/xargs.1.html https://man7.org/linux/man-pages/man1/xargs.1.html: “The command line for command is built up until it reaches a system-defined limit (unless the -n and -L options are used). The specified command will be invoked as many times as necessary to use up the list of input items. In general, there will be many fewer invocations of command than there were items in the input. This will normally have significant performance benefits.” Your only risk is that it won’t handle inputs that, on its own, are too long.
mort96 1y ago
Linking together thousands of object files with the object paths to the linker binary as command line arguments is probably the most obvious example of where the command length limit becomes problematic. Most linkers have workarounds, I think you can write the paths separated by newlines to a file and make the linker read object file paths from that file. But it would be nice if such workarounds were unnecessary.
- AlotOfReading 1y ago
  I tracked down a fun bug early in my career with those kinds of paths. The compiler driver was internally invoking a shell and had an off-by-one error that caused it to drop every 1023rd character if the total length exceeded 4k.
- sdht0 1y ago
  Yup, see https://github.com/NixOS/nixpkgs/issues/41340 https://github.com/NixOS/nixpkgs/issues/41340
- ogurechny 1y ago
  This sounds like a job for named pipes. You get the temporary file, but nothing is actually written to disk. Or maybe unnamed pipes, if bash command redirection is suitable for creating the list of options. Looking back, it's unfortunate that Unix authors offered piping of input and output streams, but did not extend that to arbitrary number of streams, making process arguments just a list of streams (with some shorthand form for constants to type in command line, and universal grammar). We could have been used to programs that react to multiple inputs or produce multiple outputs. It is obvious that it made sense in the '70s to just copy the call string to some free chunk of memory in the system record for the starting process, and let it parse those bytes in any way it wants, but, as a result, we can't just switch from list of arguments to arbitrary stream without rewriting the program. In that sense, argument strings are themselves a workaround, a quick hack which gave birth to ad-hoc serialisation rules, multi-level escaping chains, lines that are “too long” for this random system or for that random system, etc.
  - account42 1y ago
qart 1y ago
I have worked on projects (research, not production) where doing "ls" on some directories would crash the system. Some processes generated that many data files. These files had to be fed to other programs that did further processing on them. That's when I learned to use xargs.
Brian_K_White 1y ago
Backups containing other backups containing other backups containing vms/containers containing backups... all with deep paths and long path names. Balloons real fast with even just a couple arguments.

tomrod 1y ago

A huge security manifold to encourage adoption then sell white hat services on top?

dataflow 1y ago

> I don't really know what a 4k JPEG worth of command line arguments is even supposed to be used for. I didn't either, until I learned about compiler command line flags. But also: the nice thing about conmand line flags is they aren't persisted anywhere (normally). That's good for security.

eightys3v3n 1y ago
There were ways for other users on the system to see running process command line flags I thought. This isn't so good for security.
- dataflow 1y ago
  That depends on your threat model.
Ghoelian 1y ago
Pretty sure `.bash_history` includes arguments as well.
- dataflow 1y ago
  That's only if you're launching from an interactive Bash? We're talking about subprocess launches in general.
  - johnisgood 1y ago
    If you add a whitespace before the command, it will not even get appended to history!
    - amelius 1y ago
      This is quite annoying behavior, actually.
      - nosrepa 1y ago
        Good thing you can turn it off!
        amelius 1y ago
        Yeah, took me a while to figure that out though. Plus I lost my history. Typing an extra space should not invoke advanced functionality. Bad design, etc.
    - zamadatix 1y ago
      I had never run across this and it didn't work for me when I tried it. After some reading it looks like the HISTCONTROL variable needs to be set to include "ignorespace" or "ignoreboth" (the other option included in this is "ignoredups"). This would be really killer if it was always enabled and the same across shells but "some shells support something akin and you have to check if it is actually enabled on the ones that do" is just annoying enough that I probably won't bother adopting this on my local machine even though it sounds convenient as a concept.
      - wongarsu 1y ago
        I'm mostly using debian and ubuntu flavors (both on desktop and the cloud images provided and customized by various cloud and hosting providers) and they have all had this as the default behavior for bash YMMV with other shells and base distros

wongarsu 1y ago

tar cf metadata.tar.zstd *.json in a large directory of image files with json sidecars Somebody will say use a database, but when working for example with ML training data one label file per image is a common setup and what most tooling expects, and this extends further up the data preparation chain

NegativeK 1y ago
I won't say use a database, but I will beg you to compress a parent directory instead of making a tar bomb.
- mort96 1y ago
  That really just makes the problem worse: tar czf whatever.tgz whatever/*.json; you've added 9 bytes to every file path. I mean I get that you're suggesting to provide only one directory on the argv. But it sucks that the above solution to add json files to an archive while ignoring non-json files only works below some not-insane number of files.
  - stouset 1y ago
    Having an insane number of files in the same directory is already a performance killer. Here you’re talking about having something like 10,000 JSON files in one directory plus some number of non-JSON files, and you’d just be better off in all cases having these things split across separate directories. Does it suck you can’t use globing for this situation? Sure, yeah, fine, but by the time it’s a problem you’re already starting to push the limits of other parts of the system too. Also using that glob is definitely going to bite you when you forget some day that some of the files you needed were in subdirectories.
    - wongarsu 1y ago
      But in this example the files are conceptually a flat structure. Any hierarchy would be artificial, like the common ./[first two letters]/[second two letters]/filename structure. Which you can do, but it certainly doesn't make creating the above tarball any easier. Now we really need to use some kind of `find` invocation instead of a simple glob It also just extends the original question. If I have a system with 96GB RAM and terabytes of fast SSD storage, why shouldn't I be able to put tens of thousands of files in a directory and write a glob that matches half of them? I get that this was inconceivable in v6 unix, but in modern times those are entirely reasonable numbers. Heck, Windows Explorer can do that in a GUI, on a network drive. And that's a program that has been treated as essentially feature complete for nearly 30 years now, on an OS with a famously slow file system stack. Why shouldn't I be able to do the same on a linux command line?

silverwind 1y ago

There's numerous examples of useful long commands, here is one: perl -p -i -e 's#foo#bar#g' **/*

PaulDavisThe1st 1y ago
no limits: find . -type f -exec perl -p -i -e 's#foo#bar#g' {} \;
- Someone 1y ago
  That runs perl multiple times, possibly/likely often in calls that effectively are no-ops. To optimize the number of invocations of perl, you can/should use xargs (with -0)
  - dr4g0n 1y ago
    No need for `xargs` in this case, `find` has been able to take care of this for quite some time now, using `+` instead of `;`: find . -type f -exec perl -p -i -e 's#foo#bar#g' {} +
  - tremon 1y ago
    xargs constructs a command line from the find results, so if **/* exceeds the max command line length, so will xargs.
    - Someone 1y ago
      xargs was written to avoid that problem, so no, it won’t. https://man7.org/linux/man-pages/man1/xargs.1.html https://man7.org/linux/man-pages/man1/xargs.1.html: “The command line for command is built up until it reaches a system-defined limit (unless the -n and -L options are used). The specified command will be invoked as many times as necessary to use up the list of input items. In general, there will be many fewer invocations of command than there were items in the input. This will normally have significant performance benefits.” Your only risk is that it won’t handle inputs that, on its own, are too long.

mort96 1y ago

Linking together thousands of object files with the object paths to the linker binary as command line arguments is probably the most obvious example of where the command length limit becomes problematic. Most linkers have workarounds, I think you can write the paths separated by newlines to a file and make the linker read object file paths from that file. But it would be nice if such workarounds were unnecessary.

AlotOfReading 1y ago
I tracked down a fun bug early in my career with those kinds of paths. The compiler driver was internally invoking a shell and had an off-by-one error that caused it to drop every 1023rd character if the total length exceeded 4k.
sdht0 1y ago
Yup, see https://github.com/NixOS/nixpkgs/issues/41340 https://github.com/NixOS/nixpkgs/issues/41340
ogurechny 1y ago
This sounds like a job for named pipes. You get the temporary file, but nothing is actually written to disk. Or maybe unnamed pipes, if bash command redirection is suitable for creating the list of options. Looking back, it's unfortunate that Unix authors offered piping of input and output streams, but did not extend that to arbitrary number of streams, making process arguments just a list of streams (with some shorthand form for constants to type in command line, and universal grammar). We could have been used to programs that react to multiple inputs or produce multiple outputs. It is obvious that it made sense in the '70s to just copy the call string to some free chunk of memory in the system record for the starting process, and let it parse those bytes in any way it wants, but, as a result, we can't just switch from list of arguments to arbitrary stream without rewriting the program. In that sense, argument strings are themselves a workaround, a quick hack which gave birth to ad-hoc serialisation rules, multi-level escaping chains, lines that are “too long” for this random system or for that random system, etc.
- account42 1y ago

qart 1y ago

I have worked on projects (research, not production) where doing "ls" on some directories would crash the system. Some processes generated that many data files. These files had to be fed to other programs that did further processing on them. That's when I learned to use xargs.

Brian_K_White 1y ago

Backups containing other backups containing other backups containing vms/containers containing backups... all with deep paths and long path names. Balloons real fast with even just a couple arguments.