9 ms·

I think that when someone uses ls instead of a glob it means they most probably don't understand shell. I don't see any advantage of parsing ls output when glob

• DonHopkins 2y ago
  When you don't want to waste your time and sanity and happiness being in doubt and then throwing away all you've done and switching to a new language in mid stream, just don't even start using a terribly crippled shell scripting language in the first place, also and especially including awk. The tired old "stick to bash because it's already installed everywhere" argument is just as weak and misleading and pernicious as the "stick to Internet Explorer because it's already installed everywhere" argument. It's not like it isn't trivial to install Python on any system you'll encounter, unless you're programming an Analytical Engine or Jacquard Loom with punched cards.
  • cassianoleal 2y ago
    In most places where I run shell scripts, there is no Python. There could be if I really wanted it but it's generally unnecessary waste. On top of it, shell is better than Python for many things, not to mention faster. It's also, as you mentioned, ubiquitous. In the end, choose the tool that makes more sense. For me, a lot of the time, that's a shell script. Other times it may be Python, or Go, or Ruby, or any of the other tools in the box.
    • DonHopkins 2y ago
      A waste of what, disk space? I'd much rather waste a few megabytes of disk space than hours or days of my time, which is much more precious. And what are you doing on those servers, anyway? Installing huge amounts of software, I bet. So install a little more! For decades, on most Windows computers I run web browsers, there's always Internet Explorer. So do you still always use IE because installing Chrome is "wasteful"? It's a hell of a lot bigger and more wasteful than Python. As I already said, that is a weak and misleading and pernicious argument. So what exactly is bash better than Python at, besides just starting up, which only matters if you write millions of little bash and awk and sed and find and tr and jq and curl scripts that all call each other, because none of them are powerful or integrated enough to solve the problem on their own. Bash forces you to represent everything as strings, parsing and serializing and re-parsing them again and again. Even something as simple as manipulating json requires forking off a ridiculous number of processes, and parsing and serializing the JSON again and again, instead of simply keeping and manipulating it as efficient native data structures. It makes absolutely no sense to choose a tool that you know is going to hit the wall soon, so you have to throw out everything you've done and rewrite it in another language. And you don't seem to realize that when you're duct-taping together all these other half-assed languages with their quirky non-standard incompatible byzantine flourishes of command line parameters and weak antique domain specific languages, like find, awk, sed, jq, curl, etc, you're ping-ponging between many different inadequate half-assed languages, and paying the price for starting up and shutting down each of their interpreters many times over, and serializing and deserializing and escaping and unescaping their command line parameters, stdin, and stdout, which totally blows away bash's quick start-up advantage. You're arguing for learning and cobbling together a dozen or so different half-assed languages and flimsy tools, none of which you can also use to do general purpose programming, user interfaces, machine learning, web servers and clients, etc. Why learn the quirks and limitations of all those shitty complex tools, and pay the cognitive price and resource overhead of stringing them all together, when you can simply learn one tool that can do all of that much more efficiently in one process, without any quirks and limitations and duct tape, and is much easier to debug and maintain?
      • nolist_policy 2y ago
        Plus jq and curl might not even be installed. And I never got warm with jq, so if I need to parse json from shell I reach for... python. Really.
        • radiator 2y ago
          Alternatively, maybe you can get warmer with JMESPath, which has jp as its command line interface https://github.com/jmespath/jp https://github.com/jmespath/jp . The good thing about the JMESPath syntax is that it is the standard one when processing JSON in software like Ansible, Grafana, perhaps some more.
        • mywittyname 2y ago
          I'm an avid jq user. There are certainly situations where it's better to use python because it's just more sane and easier to read/write, but jq does a few things extremely well, namely, compressing json, and converting json files consisting of big-ass arrays into line delimited json files.
      • cassianoleal 2y ago
        > For decades, on most Windows computers I run web browsers, there's always Internet Explorer. As I already said, that is a weak and misleading and pernicious argument. On its own, I agree. But you glossed over everything else I said, so I'm not going to entertain your weak argument. You seem to ignore that different users, different use cases, different environments, etc. all need to be taken into account when choosing a tool. Like I said, for most of my use cases where I use shell scripting, it's the best tool for the job. If you don't believe me, or think you know better about my circumstances than I do, all the power to you.
        • sqeaky 2y ago
          > You seem to ignore that different users, different use cases, different environments, etc. all need to be taken into account when choosing a tool. I have worked on projects that are extremely sensitive to extra dependencies and projects that aren't. Sometimes I am in an underground bunker and each dependency goes through an 18 month Department of Defense vetting process, and "Just install python" is equivalent to "just don't do the project". Other times I have worked on projects where tech debt was an afterthought because we didn't know if the code would still be around in a week and re-writing was a real option, so bringing in a dependency for a single command was worthwhile if we could solve the problem now. There is appetite for risk, desire for control, need for flexibility, and many other factors just as you stated that DonHopkins is ignoring or unaware of.
  • deleted 2y ago
    [deleted]
• Pesthuf 2y ago
  People new to *nix make the mistake of thinking this stuff is well designed, makes sense and that things work well together. They learn. We all do.
  • larodi 2y ago
    People new to the internet think alike. Still, not a day passes and we are once again reminded how fragile yet amazing this all information theory stuff is.
  • lukan 2y ago
    Coincidently I discovered the unix haters handbook today: https://web.mit.edu/~simsong/www/ugh.pdf https://web.mit.edu/~simsong/www/ugh.pdf
    • pikzel 2y ago
      "The Macintosh on which I type this has 64MB: Unix was not designed for the Mac. What kind of challenge is there when you have that much RAM?" Love it.
      • Narishma 2y ago
        I don't understand what they mean in that quote. Neither Unix nor the Mac were designed for that much RAM.
        • II2II 2y ago
          Judging from the context, the user interface was fine in the days of limited resources (a 16 kiloword PDP-11 was cited) but then modern computers have the resources for better user interfaces. They clearly didn't realize that even more modern Unix kernels would require hundreds of megabytes just to boot.
          • paulddraper 2y ago
            What kernel takes 200 MB+ to boot?
• CJefferson 2y ago
  Sometimes I want all filenames from a subdirectory, without the subdirectory name. I can do (ignoring parsing issues): for name in $(cd subdir; ls); do echo "$name"; done This isn't easy to do with globbing (as far as I know)
  • rascul 2y ago
    One alternative: for name in subdir/*; do basename "$name"; done
    • Izkata 2y ago
      Also since subdir is hardcoded, you can reliably type it a second time to chop off however much of the start you want: for name in subdir/subsubdir/*; do echo "${name#subdir/}" # subsubdir/foo done
      • xolox 2y ago
        Note this string replacement is not anchored (right?) which can end up biting you badly (depending on circumstances of course).
        • chuckadams 2y ago
          It's anchored on the left. ${name#subdir/} will turn 'subdir/abc' into 'abc', but will not touch foo/subdir/bar. I don't think bash even has syntax to replace in the middle of an expansion, I always pull out sed for that.
          • xolox 2y ago
            Thanks for clarifying, I learned something new today! Edit: It turns out that Bash does substitutions in the middle of strings using the ${string/substring/replacement} and ${string//substring/replacement} syntax, for more details see https://tldp.org/LDP/abs/html/string-manipulation.html https://tldp.org/LDP/abs/html/string-manipulation.html
• englishspot 2y ago
  I just use find. it's a little longer but gives me the full paths and is more consistent. also works well if you need to recurse. something like `find . -type f | while read -r filepath; do whatever "${filepath}"; done`
  • gnuvince 2y ago
    Is there a reason to prefer `while read; ...;done` over find's -exec or piping into xargs?
    • xolox 2y ago
      Both `find -exec` and xargs expect an executable command whereas `while read; ...; done` executes inline shell code. Of course you can pass `sh -c '...'` (or Bash or $SHELL) to `find -exec` or xargs but then you easily get into quoting hell for anything non-trivial, especially if you need to share state from the parent process to the (grand) child process. You can actually get `find -exec` and xargs to execute a function defined in the parent shell script (the one that's running the `find -exec` or xargs child process) using `export -f` but to me this feels like a somewhat obscure use case versus just using an inline while loop.
    • retrogeek 2y ago
      I will sometimes use the "| while read" syntax with find. One reason for doing so is that the "-exec" option to find uses {} to represent the found path, and it can only be used ONCE. Sometimes I need to use the found path more than once in what I'm executing, and capturing it via a read into a reusable variable is the easiest option for that. I'd say I use "-exec" and "| while read" about equally, actually. And I admittedly almost NEVER use xargs.
  • ykonstant 2y ago
    This will fail for files with newlines.
    • michaeldh 2y ago
      How common are they?
• dotancohen 2y ago
  > I think that when someone uses ls instead of a glob it means they most probably don't understand shell. In 25 years of using Bash, I've picked up the knowledge that I shouldn't parse the output of ls. I suppose that it has something to do with spaces, newlines, and non-printing characters in file names. I really don't know. But I do know that when I'm scripting, I'm generally wrapping what I do by hand, in a file. I'm codifying my decisions with ifs and such, but I'm using the same tools that I use by hand. And ls is the only tool that I use to list files by hand - so I find it natural that people would (naively) pick ls as the tool to do that in scripts.
  • notnmeyer 2y ago
    exactly—well said
  • AtlasBarfed 2y ago
    What I don't understand is why in the two most popular Unix flavors we have not got something like a json list output or something else that is parsable. Is it really that difficult to add --json as a flag?
    • out-of-ideas 2y ago
      i think it boils down to then is dependencies needed to parse the json, coupled with the fact that glob syntax already covers iterating over the files regardless of characters used in the filename. there are other tools than `ls` with their soul purpose to list files; some have "improved" features than ls, ect similarlly from above (and well said btw, even what is not quoted): > I do know that when I'm scripting, I'm generally wrapping what I do by hand, in a file. I'm codifying my decisions with ifs and such, but I'm using the same tools that I use by hand a lot of us do similar and we know/expect the ins and outs; and all it takes to break our scripts is some edge case we never thought of. we are fortunate that our keyboard layouts are basically ascii; other languages are less fortunate. now introduce open source community driven software where an ls escaped bash code deletes somebodys home directory (as ls was parsed and an edge case of some users files cause some obscure "fun" times). an edge case is still painful.. and finally, sometimes its better elevating said bash script to python (or awk), ect. just depends on the situation and level of complexity of logic
• PaulHoule 2y ago
  I went through a phase when I really enjoyed writing shell scripts like ls *.jpg | awk '{print "resize 200x200 $1 thumbnails/$1"}' | bash because I never got to the point where I could remember the strange punctuation that the shell requires for loops without looking up the info pages for bash whereas I've thoroughly internalized awk syntax. Word is you should never write something like that because you'll never get the escaping right and somebody could craft inputs that would cause arbitrary code execution. I mean, they try to scare you into using xargs, but I find xargs so foreign I have to read the whole man page every time I want to do something with it.
  • hifromwork 2y ago
    I encourage you to give it a try again. Almost every use of xargs that I ever did looked like this: ls *.jpg | xargs -i,, resize 200x200 ,, thumbnails/,, I just always define the placeholder to ,, (you can pick something else but ,, is nice and unique) and write commands like you do.
    • kstrauser 2y ago
      I'm more likely to write that like: for i in *.jpg; resize 200x200 "$i" "thumbnails/$i"; end
      • TylerE 2y ago
        Does that not fail when you hit the maximum command line length? Doesn't the entirety of the directory get splatted? Isn't this the whole reason xargs exists?
        • PaulHoule 2y ago
          Exactly, there are so many limits in the shell that I don’t want to be bothered to think about. When I get serious I just write Python.
        • genrilz 2y ago
          The for loop only runs resize once per file. So no, the entire directory does not get splatted. It is unlikely you'd hit maximum command length. At least on mac, the max command length is 1048576 bytes, while the maximum path length in the home directory is 1024 bytes. There might be some unix variant where the max path length is close enough to the max command length to cause an overflow, but I doubt that is the case for common ones. xargs exists in an attempt to be able to parse command output. You could for instance have awk output xargs formatted file names to build up a single command invocation from arbitrary records read by awk. Note that xargs still has to obey the command line length limit though, because the command line needs to get passed to the program. Thus, in a situation where this for loop overflows the command line, it would cause xargs to also fail. Thus I would always use globbing if I have the choice. EDIT: If you mean that the directory is splatted in the for loop, then in a theoretical sense it is. However, since "for" is a shell builtin, it does not have to care about command line length limits to my knowledge.
• chrsig 2y ago
  Commands can have a maximum number of arguments. Try globbing on a directory with millions of files.
  • anthk 2y ago
    Sane people will just use find and/or xargs.
    • nmz 2y ago
      Weird thing to call sane when its the shell that is insane, or more likely an instrument of torture.
  • arp242 2y ago
    Usually the pattern is "for f in [glob]", which doesn't have that issue. Running "ls" on a directory is little more than "for f in *; echo $f" so there's little advantage to using "ls". Also: "find -exec {} \+" will take ARG_MAX into account, and may be much faster depending on what you're doing.
• stephenr 2y ago
  One advantage: `ls -i` gives you the file's inode in a POSIX portable way. If you glob and then look it up individually for each file, you'll need to be aware of which tool (and whether it's GNU or BSD in origin) you use on which platform. In general yes globbing is better for iterating through files. But parsing `ls` doesn't necessarily mean the author doesn't know shell. It might mean they know it well enough to use the tools that are made available to them.
• seanhunter 2y ago
  It's not the case very often these days but it used to be quite simple to blow up your script globbing in a directory with a lot of files and you can still hit the limit if you pass a glob to some command because it can blow up trying to execve() Here's more details of the issue and some workarounds https://unix.stackexchange.com/questions/120642/what-defines-the-maximum-size-for-a-command-single-argument https://unix.stackexchange.com/questions/120642/what-defines...