5 ms·

AND YET LS STILL IS NOT RELIABLY PARSEABLE And yet there aren't --json (and/or --yaml) universal output option flags to structure your data in a reliably parse

AND YET LS STILL IS NOT RELIABLY PARSEABLE And yet there aren't --json (and/or --yaml) universal output option flags to structure your data in a reliably parseable format. Shouldn't every option for a major UNIX command (insert your definition for major) have an extensive list of useful examples/use cases? Man still fails me here. As does the AWS CLI too as a side complaint. If your option is important and useful, then it demands a good example.

munk-a 5y ago
I read in an article (this one - joke) that all lower case letters except `jvyz` are in use - it looks like `-j` and `-y` are just sitting there waiting for someone to add some output formats - then we just need to add a verbose flag and... maybe a switch that automatically gzips the output? We're so close to the 100% completion achievement - we can get there in our lifetimes.
shatteredgate 5y ago
It doesn't really make sense to have unix commands output json or yaml. They're already doing a form of parsing internally, you're then asking them to encode the output in another text format so you can parse it again. Why? You'd have better results just using the underlying API to get a list of files, and this is what every other program that needs to get a list of files actually does. From a scripting perspective this is one of the things that powershell actually got right.
- AtlasBarfed 5y ago
  well, because, ls, lsblk, ps, and a host of other table-based outputs aren't reliably parseable. Their columns overflow. They aren't durable to line breaks in indentifiers and other UTF8 and control character oddities. You know what is? A decent data output format. And JSON is the best of the compromises that exist now. YAML would be nice because if you are outputting json, yaml output is easy and it's a bit more readable. That doesn't violate the UNIX philosophy of text output. It probably improves it. Underlying APIs aren't as universal, portable, or well-known. We are talking basically GNU cross-platform programs.
  - shatteredgate 5y ago
    But you don't actually want to parse them though, that's the mechanism by which you receive the data but the real end goal is you just want the data in its canonical format so it can used programmatically; in powershell there is no parsing, the output is always the canonical representation. I don't get what you meant in the last sentence, the underlying API is glibc which is also cross platform and is incredibly well known. Other cross-platform languages like Java, Golang, Rust, etc have their own portable APIs to enumerate directories that are cross platform.
    - AtlasBarfed 5y ago
      Perl. Python. Bash or shell scripts. They all have "invoke this shell command" at a minimum. Then you get into ssh and a host of other command invocations to get information. The shell is the convenient way to invoke commands to get the information: locally, remotely, quickly, no setup of C bindings, lots of invocation options. If you are polyglot, do you want to use a command you know (ls) or learn each language's infernal syntax for filtering, recursive listing, etc, or learn it once with shell?
      - shatteredgate 5y ago
        >If you are polyglot, do you want to use a command you know (ls) or learn each language's infernal syntax for filtering, recursive listing, etc, or learn it once with shell? Probably learn the language's syntax because it's faster and more powerful, e.g. in python and perl I'd say don't bother with running shell commands and doing unnecessary forking and parsing if you can avoid it, use the built-in glob instead: https://docs.python.org/3/library/glob.html https://docs.python.org/3/library/glob.html https://perldoc.perl.org/functions/glob https://perldoc.perl.org/functions/glob Most languages just have this now because it's easier and faster than invoking external programs and is a better cross platform solution, glibc even has it: https://man7.org/linux/man-pages/man3/glob.3.html https://man7.org/linux/man-pages/man3/glob.3.html
justinsaccount 5y ago
Something like https://github.com/Juniper/libxo https://github.com/Juniper/libxo is probably the best bet for non-powershell, non-python like things to ever be properly parseable.
trasz 5y ago
There are, to some extent, in FreeBSD - you can eg do "vmstat --libxo=json". Not for ls(1), though - I think it's just easier to list files using facilities provided by your language of choice than running external tool and parsing JSON.

munk-a 5y ago

I read in an article (this one - joke) that all lower case letters except `jvyz` are in use - it looks like `-j` and `-y` are just sitting there waiting for someone to add some output formats - then we just need to add a verbose flag and... maybe a switch that automatically gzips the output? We're so close to the 100% completion achievement - we can get there in our lifetimes.

shatteredgate 5y ago

It doesn't really make sense to have unix commands output json or yaml. They're already doing a form of parsing internally, you're then asking them to encode the output in another text format so you can parse it again. Why? You'd have better results just using the underlying API to get a list of files, and this is what every other program that needs to get a list of files actually does. From a scripting perspective this is one of the things that powershell actually got right.

AtlasBarfed 5y ago
well, because, ls, lsblk, ps, and a host of other table-based outputs aren't reliably parseable. Their columns overflow. They aren't durable to line breaks in indentifiers and other UTF8 and control character oddities. You know what is? A decent data output format. And JSON is the best of the compromises that exist now. YAML would be nice because if you are outputting json, yaml output is easy and it's a bit more readable. That doesn't violate the UNIX philosophy of text output. It probably improves it. Underlying APIs aren't as universal, portable, or well-known. We are talking basically GNU cross-platform programs.
- shatteredgate 5y ago
  But you don't actually want to parse them though, that's the mechanism by which you receive the data but the real end goal is you just want the data in its canonical format so it can used programmatically; in powershell there is no parsing, the output is always the canonical representation. I don't get what you meant in the last sentence, the underlying API is glibc which is also cross platform and is incredibly well known. Other cross-platform languages like Java, Golang, Rust, etc have their own portable APIs to enumerate directories that are cross platform.
  - AtlasBarfed 5y ago
    Perl. Python. Bash or shell scripts. They all have "invoke this shell command" at a minimum. Then you get into ssh and a host of other command invocations to get information. The shell is the convenient way to invoke commands to get the information: locally, remotely, quickly, no setup of C bindings, lots of invocation options. If you are polyglot, do you want to use a command you know (ls) or learn each language's infernal syntax for filtering, recursive listing, etc, or learn it once with shell?
    - shatteredgate 5y ago
      >If you are polyglot, do you want to use a command you know (ls) or learn each language's infernal syntax for filtering, recursive listing, etc, or learn it once with shell? Probably learn the language's syntax because it's faster and more powerful, e.g. in python and perl I'd say don't bother with running shell commands and doing unnecessary forking and parsing if you can avoid it, use the built-in glob instead: https://docs.python.org/3/library/glob.html https://docs.python.org/3/library/glob.html https://perldoc.perl.org/functions/glob https://perldoc.perl.org/functions/glob Most languages just have this now because it's easier and faster than invoking external programs and is a better cross platform solution, glibc even has it: https://man7.org/linux/man-pages/man3/glob.3.html https://man7.org/linux/man-pages/man3/glob.3.html

justinsaccount 5y ago

Something like https://github.com/Juniper/libxo https://github.com/Juniper/libxo is probably the best bet for non-powershell, non-python like things to ever be properly parseable.

trasz 5y ago

There are, to some extent, in FreeBSD - you can eg do "vmstat --libxo=json". Not for ls(1), though - I think it's just easier to list files using facilities provided by your language of choice than running external tool and parsing JSON.