5 ms·

In a certain sense, files everywhere is great, that's the promise of unix, or plan9 to a further extent. However, unstructured files, or files that all have th

In a certain sense, files everywhere is great, that's the promise of unix, or plan9 to a further extent. However, unstructured files, or files that all have their own formats, is also equally hampering. Trying to even parse an nginx log file can be annoying with just awk or some such. One of the big disadvantages is that large system rewrites and design changes cannot be executed in the linux userland. All to say, I'd love a smarter shell, I love files, I have my awk book sitting next to me, but I think it's high time to get some serious improvements on parsing data. In the same way programs are smart enough to know to render colored output or not, I'd love it if it could dump structured output (or not)

mikepurvis 3y ago
When it comes to parsing server logs, it's too bad the functionality can't be extracted out of something like logstash, since that's already basically doing the same thing. Though I guess the real endgame here is for upstream tools to eventually recognize the value and learn how to directly supply structured output.
- dale_glass 3y ago
  You can get that out of journald. journalctl -o json And applications using journald directly can provide their own custom fields.
hoherd 3y ago
> In the same way programs are smart enough to know to render colored output or not, I'd love it if it could dump structured output (or not) The even lower hanging fruit is to implement json output as a command line argument in all cli tools. I would love to see this done for the gnu core utils.
- teddyh 3y ago
  At least “du”, “env”, and “printenv” in Coreutils all support the “--null” option to separate output into being NUL-separated instead of being text lines.
- bryanlarsen 3y ago
  It was a really pleasant surprise to find the "-j" option to do this for the "ip" command from the iproute2 project.
- mr_mitm 3y ago
  AFAIK the idea is that if you need that kind of interoperability in unix/gnu, you're supposed to write your tool in C and include some libraries. Clearly not realistic in many use cases.
dale_glass 3y ago
Yup. It really grinds my gears that people came up with fairly decent ideas half a century ago, and a large amount of people decided to take that as gospel rather than as something to improve on. And it's like pulling teeth to get any improvement, because the moment somebody like Lennart tries to get rid of decades of old cruft, drama erupts. And even JSON is still not quite there. JSON is an okay-ish idea, but to do this properly what we need is a format that can expose things like datatypes. More like PowerShell. So that we can do amazing feats like treating a number like a number, and calculating differences between dates by doing $a - $b.
- CyberDildonics 3y ago
  If you want something more complex and restrictive you could easily make it out of JSON. JSON works because it is simple and isn't being constantly distorted into something more complicated to cover niche use cases.
- zlg_codes 3y ago
  Yeah dude I totally love it when journald eats my logging and power interruptions create unrecoverable log loss! You can forget just mounting a drive and checking its journald logs -- your machine has to be started by systemd in order to read journald logs! I totally love it when liars frontmanning the project say it's a project and an init system and a system layer that will replace everything. But it won't! Pay no attention to the inconsistent messaging, the "gentle pushes" to get other distros to use it, etc. Also, let's not mention he basically hoodwinked the entire community by switching teams to Microsoft. And people like you eat his work up! Are you sure you like free software?
  - dale_glass 3y ago
    > Yeah dude I totally love it when journald eats my logging and power interruptions create unrecoverable log loss! So does everything else, because of consumer drives. Fun fact: filesystems by default only guarantee the integrity of the filesystem's structure itself. The promise is that after power loss, the basic structures of the filesystem won't be corrupt, but makes no big claims of reliability about the data written. Blocks of random junk in your data, blocks of NULLs, even parts of other files (maybe deleted data) are all things I've seen happen. Databases and the like take serious effort to ensure safety, but that greatly slows down performance. So it's kind of a hard sell for a log system that's not supposed to be a performance impact. For this contingency, you can do log shipping, but in general, system logs shouldn't be expected to be reliable in the face of a crash, since to my knowledge all normal daemons (including rsyslogd) buffer data before flushing to disk. > You can forget just mounting a drive and checking its journald logs -- your machine has to be started by systemd in order to read journald logs! No, it doesn't. There are file arguments to journalctl. Read the manpage, sheesh. > I totally love it when liars frontmanning the project say it's a project and an init system and a system layer that will replace everything. But it won't! Pay no attention to the inconsistent messaging, the "gentle pushes" to get other distros to use it, etc. Meh. Paranoia. > Also, let's not mention he basically hoodwinked the entire community by switching teams to Microsoft. And people like you eat his work up! Are you sure you like free software? Free Software is a licensing/distribution concept completely unrelated to whether one likes or not Microsoft's technical decisions. Some I really hate, and some are actually pretty cool. I'm for Free Software because I like the licensing philosophy, not because I believe Unix is the best thing since sliced bread. In fact I believe Unix started as a bunch of good ideas but that have not kept up and so needs a bit of work to remain a good system to use.
da_chicken 3y ago
There is always Powershell. The trouble there is that it's so rooted in .Net and objects that it's very difficult to integrate with existing native commands on any platform.
- uxp8u61q 3y ago
  If these "native commands" had a sensible output format, integrating them with powershell would be as simple as putting ConvertFrom-Json or ConvertFrom-Csv in the middle of the pipeline. And let's be real, it's as poorly "integrated" with "native commands" as bash or zsh is poorly integrated with native commands.
  - da_chicken 3y ago
    You're not wrong, but the fix is then, "hey, let's extend the functionality of literally every *nix command". Which is hard to achieve. Building sensible serialized object-oriented output is the same problem as building sensible object-oriented output. That's why jc exists, I suppose. There is always ConvertFrom-String. The problem is that it is one of the least intuitive and worst-performing commands I've used in Powershell. It's awful and I hate it. It's like writing sed and awk commands without the benefit of sed and awk's maturity and ubiquity. IMX, only Compare-Object has been worse.
    - uxp8u61q 3y ago
      > "hey, let's extend the functionality of literally every *nix command". Which is hard to achieve. It's pretty much what powershell did, though.
      - imtringued 3y ago
        It sounds like there needs to be a monolithic solution built from the ground up for interoperation. Updating coreutils is a losing game at this point. Of course people will get angry when a well designed solution gradually takes over simply, because it is better.
mistercow 3y ago
Part of the problem is that the output of commands is both a UI and an API, and because any text UI can be used as an API, the human readable text gets priority. Shell scripting is therefore kind of like building third party browser extensions. You look and you guess, and then you hack some parser up based on your guess, and hope for the best. I actually wish there was just a third standard output for machine readable content, which your terminal doesn’t print by default. When you pipe, this output is what gets piped (unless you redirect), it’s expected to be jsonl, and the man page is expected to specify a contract. Then stdout can be for humans, and while you can parse it, you know what you’re doing is fragile. Of course, that’s totally backwards incompatible, and as long as we’re unrealistically reinventing CLIs from the foundations to modernize them, I have a long list of changes I’d make.
- theblazehen 3y ago
  Have you seen some of the existing projects currently working on it? Most well known is https://www.nushell.sh/ https://www.nushell.sh/, amongst some others
- chongli 3y ago
  I really want to agree because it seems to make so much sense in theory. It gets rid of the need to parse by standardizing on syntax. But it doesn't solve everything, so we still don't get to live the dream. Namely, it does not solve the issue of schemas, versioning, and migration. And this is a really big issue that threatens to derail the whole project. If my script runs a pipeline `a | b | c` and utility b gets updated with a breaking change to the schema, it breaks the entire script. Now I've got to go deep into the weeds to figure out what happened, and the breakage might not be visible in the human-readable output. So to debug I'll have to pass the flag to each tool to get it to print all the json to stdout, and then sit there eyeballing the json to figure out how the schema changed and what I need to do to fix my script. Seems like a big mess to me. Unless there's something I'm missing?
  - kbrazil 3y ago
    There could be a major schema change that breaks the contract, but one of the nice things about JSON output is that it allows the creation of new fields without affecting downstream consumers. That is, if I have a CLI program that spits out a list of IP addresses and one day I want to also output the corresponding dns names, I can simply add the "dns" field and existing pipelines will ignore the field and work just fine. This is better than grep/awking/etc. unstructured text to STDOUT because, depending on how the author decides to add the new field, it can easily break existing pipelines that rely on the shape of the data to stay the same.
numbsafari 3y ago
> Trying to even parse an nginx log file can be annoying with just awk or some such. You probably already know this, but for those who do not, you can configure nginx to generate JSON log output. Quite handy if you are aggregating structured logs across your stack.
imtringued 3y ago
The problem is that nobody has built an actual ffi solution except maybe the GObject guys. C isn't an ffi, because you need a C compiler to make it work. By that I mean it is not an interface, but rather just C code whose calling part has been embedded into your application.

mikepurvis 3y ago

When it comes to parsing server logs, it's too bad the functionality can't be extracted out of something like logstash, since that's already basically doing the same thing. Though I guess the real endgame here is for upstream tools to eventually recognize the value and learn how to directly supply structured output.

dale_glass 3y ago
You can get that out of journald. journalctl -o json And applications using journald directly can provide their own custom fields.

hoherd 3y ago

> In the same way programs are smart enough to know to render colored output or not, I'd love it if it could dump structured output (or not) The even lower hanging fruit is to implement json output as a command line argument in all cli tools. I would love to see this done for the gnu core utils.

teddyh 3y ago
At least “du”, “env”, and “printenv” in Coreutils all support the “--null” option to separate output into being NUL-separated instead of being text lines.
bryanlarsen 3y ago
It was a really pleasant surprise to find the "-j" option to do this for the "ip" command from the iproute2 project.
mr_mitm 3y ago
AFAIK the idea is that if you need that kind of interoperability in unix/gnu, you're supposed to write your tool in C and include some libraries. Clearly not realistic in many use cases.

dale_glass 3y ago

Yup. It really grinds my gears that people came up with fairly decent ideas half a century ago, and a large amount of people decided to take that as gospel rather than as something to improve on. And it's like pulling teeth to get any improvement, because the moment somebody like Lennart tries to get rid of decades of old cruft, drama erupts. And even JSON is still not quite there. JSON is an okay-ish idea, but to do this properly what we need is a format that can expose things like datatypes. More like PowerShell. So that we can do amazing feats like treating a number like a number, and calculating differences between dates by doing $a - $b.

CyberDildonics 3y ago
If you want something more complex and restrictive you could easily make it out of JSON. JSON works because it is simple and isn't being constantly distorted into something more complicated to cover niche use cases.
zlg_codes 3y ago
Yeah dude I totally love it when journald eats my logging and power interruptions create unrecoverable log loss! You can forget just mounting a drive and checking its journald logs -- your machine has to be started by systemd in order to read journald logs! I totally love it when liars frontmanning the project say it's a project and an init system and a system layer that will replace everything. But it won't! Pay no attention to the inconsistent messaging, the "gentle pushes" to get other distros to use it, etc. Also, let's not mention he basically hoodwinked the entire community by switching teams to Microsoft. And people like you eat his work up! Are you sure you like free software?
- dale_glass 3y ago
  > Yeah dude I totally love it when journald eats my logging and power interruptions create unrecoverable log loss! So does everything else, because of consumer drives. Fun fact: filesystems by default only guarantee the integrity of the filesystem's structure itself. The promise is that after power loss, the basic structures of the filesystem won't be corrupt, but makes no big claims of reliability about the data written. Blocks of random junk in your data, blocks of NULLs, even parts of other files (maybe deleted data) are all things I've seen happen. Databases and the like take serious effort to ensure safety, but that greatly slows down performance. So it's kind of a hard sell for a log system that's not supposed to be a performance impact. For this contingency, you can do log shipping, but in general, system logs shouldn't be expected to be reliable in the face of a crash, since to my knowledge all normal daemons (including rsyslogd) buffer data before flushing to disk. > You can forget just mounting a drive and checking its journald logs -- your machine has to be started by systemd in order to read journald logs! No, it doesn't. There are file arguments to journalctl. Read the manpage, sheesh. > I totally love it when liars frontmanning the project say it's a project and an init system and a system layer that will replace everything. But it won't! Pay no attention to the inconsistent messaging, the "gentle pushes" to get other distros to use it, etc. Meh. Paranoia. > Also, let's not mention he basically hoodwinked the entire community by switching teams to Microsoft. And people like you eat his work up! Are you sure you like free software? Free Software is a licensing/distribution concept completely unrelated to whether one likes or not Microsoft's technical decisions. Some I really hate, and some are actually pretty cool. I'm for Free Software because I like the licensing philosophy, not because I believe Unix is the best thing since sliced bread. In fact I believe Unix started as a bunch of good ideas but that have not kept up and so needs a bit of work to remain a good system to use.

da_chicken 3y ago

There is always Powershell. The trouble there is that it's so rooted in .Net and objects that it's very difficult to integrate with existing native commands on any platform.

uxp8u61q 3y ago
If these "native commands" had a sensible output format, integrating them with powershell would be as simple as putting ConvertFrom-Json or ConvertFrom-Csv in the middle of the pipeline. And let's be real, it's as poorly "integrated" with "native commands" as bash or zsh is poorly integrated with native commands.
- da_chicken 3y ago
  You're not wrong, but the fix is then, "hey, let's extend the functionality of literally every *nix command". Which is hard to achieve. Building sensible serialized object-oriented output is the same problem as building sensible object-oriented output. That's why jc exists, I suppose. There is always ConvertFrom-String. The problem is that it is one of the least intuitive and worst-performing commands I've used in Powershell. It's awful and I hate it. It's like writing sed and awk commands without the benefit of sed and awk's maturity and ubiquity. IMX, only Compare-Object has been worse.
  - uxp8u61q 3y ago
    > "hey, let's extend the functionality of literally every *nix command". Which is hard to achieve. It's pretty much what powershell did, though.
    - imtringued 3y ago
      It sounds like there needs to be a monolithic solution built from the ground up for interoperation. Updating coreutils is a losing game at this point. Of course people will get angry when a well designed solution gradually takes over simply, because it is better.

mistercow 3y ago

Part of the problem is that the output of commands is both a UI and an API, and because any text UI can be used as an API, the human readable text gets priority. Shell scripting is therefore kind of like building third party browser extensions. You look and you guess, and then you hack some parser up based on your guess, and hope for the best. I actually wish there was just a third standard output for machine readable content, which your terminal doesn’t print by default. When you pipe, this output is what gets piped (unless you redirect), it’s expected to be jsonl, and the man page is expected to specify a contract. Then stdout can be for humans, and while you can parse it, you know what you’re doing is fragile. Of course, that’s totally backwards incompatible, and as long as we’re unrealistically reinventing CLIs from the foundations to modernize them, I have a long list of changes I’d make.

theblazehen 3y ago
Have you seen some of the existing projects currently working on it? Most well known is https://www.nushell.sh/ https://www.nushell.sh/, amongst some others
chongli 3y ago
I really want to agree because it seems to make so much sense in theory. It gets rid of the need to parse by standardizing on syntax. But it doesn't solve everything, so we still don't get to live the dream. Namely, it does not solve the issue of schemas, versioning, and migration. And this is a really big issue that threatens to derail the whole project. If my script runs a pipeline `a | b | c` and utility b gets updated with a breaking change to the schema, it breaks the entire script. Now I've got to go deep into the weeds to figure out what happened, and the breakage might not be visible in the human-readable output. So to debug I'll have to pass the flag to each tool to get it to print all the json to stdout, and then sit there eyeballing the json to figure out how the schema changed and what I need to do to fix my script. Seems like a big mess to me. Unless there's something I'm missing?
- kbrazil 3y ago
  There could be a major schema change that breaks the contract, but one of the nice things about JSON output is that it allows the creation of new fields without affecting downstream consumers. That is, if I have a CLI program that spits out a list of IP addresses and one day I want to also output the corresponding dns names, I can simply add the "dns" field and existing pipelines will ignore the field and work just fine. This is better than grep/awking/etc. unstructured text to STDOUT because, depending on how the author decides to add the new field, it can easily break existing pipelines that rely on the shape of the data to stay the same.

numbsafari 3y ago

> Trying to even parse an nginx log file can be annoying with just awk or some such. You probably already know this, but for those who do not, you can configure nginx to generate JSON log output. Quite handy if you are aggregating structured logs across your stack.

imtringued 3y ago

The problem is that nobody has built an actual ffi solution except maybe the GObject guys. C isn't an ffi, because you need a C compiler to make it work. By that I mean it is not an interface, but rather just C code whose calling part has been embedded into your application.