5 ms·

"With -fdiagnostics-format=json, the diagnostics are emitted as a big blob of JSON to stderr"

by lrsjng 8y ago

codetrotter 8y ago
I’ve previously been thinking about what would be the consequences of implementing a subset of Unix tools to work with a binary protocol. For example, tables would be output with type info and proper unambiguous delimiting of some sort. Unrelated to that I once wrote a program in the past several years ago as two distinct parts that communicated over a pair of Unix pipes; one part the GUI which embedded a web view and the other part which contained the program logic and which output HTML to stdout. The code was quite ugly, but anyway it did the job at the time. But the idea of outputting JSON is somewhat in between this two things. But the main takeaway here IMO is the idea about outputting a specific format as controlled by a flag. Of course other tools have done this too in the past. Consider a few Unix utilities reimplemented to work a bit tighter together in terms of how some data is represented, and using for example Cap’n Proto to serialize the data and piping that between them but also writing a library for destructively converting to plain text representation and then using some method or combination of methods for deciding when to use binary piping. For example, - A user-provided flag. Tiresome to type. - A user-provided environment variable. Problematic when mixing tools that implement serialization with pure plaintext tools. - The shell could be aware of which programs implement this serialization and invoke them with the flags or env vars corresponding to whether or not each program on each side of each pipe in a pipeline supports the serialization. Possibly the way to go. In this way one could incrementally rebuild the Unix tools to work this way without having to change everything at once and still being compatible with the universal interface of plaintext forever, since no-one wants nor can implement this in every Unix tool out there. Once you have a couple of tools support this then the possibility for new ways to interact with pipelines becomes available. I still love and will use the classic command-line. But in some situations the possibilities enabled by this would be very compelling indeed.
- Longhanks 8y ago
  What you describe is PowerShell. It is available for most Linux and macOS.
  - codetrotter 8y ago
    I am aware that PowerShell is somewhat like this — in fact I started using PS before permanently switching to Linux back in 2009 — but I don’t enjoy PS. Some people like PS a lot I know and that’s great for them but for me it’s not something I am ever going to use again. I found my home with Unix those 10 years ago.
  - pantalaimon 8y ago
    IMHO what kills PowerShell is that it's so incredibly verbose.
    - svnpenn 8y ago
      Even with the built-in aliases?
  - int_19h 8y ago
    Not quite. PowerShell input and output are .NET objects - not just data, but also behavior. This requires both sides of the pipe to agree on the object model and the execution environment. JSON (or another pure data serialization format) is much better in that regard, because then you only need to agree on some basic data structures at the boundary.
- emmelaich 8y ago
  I think csv (with a separator of your choice) would be ideal. It wouldn't require special tools but it could work with them. You could save the column headers, types and separator in extended attributes.
  - emmelaich 8y ago
    For extra points you could guarantee the structure of data by only allowing certain tools (selinux context maybe) that are trusted to write the files with the required structure. That would save a lot of repeated wasteful parsing and checking.
- caf 8y ago
  An alternative that I have had kicking around in the back of my mind for a few years is having the process on the read side of the pipe call fcntl() with a new flag to set an "Accept" set of formats, which the process on the write side of the pipe can retrieve with a 'get' equivalent. If there's a format both sides recognise, they can use that. Both sides just default to plain text so it'll be backwards compatible.
- int_19h 8y ago
  FreeBSD is using this in its base system (although I don't think it's all or even most apps as yet). http://juniper.github.io/libxo/libxo-manual.html http://juniper.github.io/libxo/libxo-manual.html
- chubot 8y ago
  I wrote a couple articles about structured data over pipes, which may motivate some features in the Oil shell [1] How to Quickly and Correctly Generate a Git Log in HTML http://www.oilshell.org/blog/2017/09/19.html http://www.oilshell.org/blog/2017/09/19.html Git Log in HTML: A Harder Problem and A Safe Solution http://www.oilshell.org/blog/2017/09/29.html http://www.oilshell.org/blog/2017/09/29.html I'm interested in ideas for structured data over pipes, or even better a prototype and example programs :) [1] http://www.oilshell.org/blog/2018/01/28.html http://www.oilshell.org/blog/2018/01/28.html
- wilde 8y ago
  This is very similar to PowerShell. But CMD.EXE just kills it for me.

"With -fdiagnostics-format=json, the diagnostics are emitted as a big blob of JSON to stderr"

by lrsjng 8y ago

"With -fdiagnostics-format=json, the diagnostics are emitted as a big blob of JSON to stderr"

I’ve previously been thinking about what would be the consequences of implementing a subset of Unix tools to work with a binary protocol. For example, tables would be output with type info and proper unambiguous delimiting of some sort. Unrelated to that I once wrote a program in the past several years ago as two distinct parts that communicated over a pair of Unix pipes; one part the GUI which embedded a web view and the other part which contained the program logic and which output HTML to stdout. The code was quite ugly, but anyway it did the job at the time. But the idea of outputting JSON is somewhat in between this two things. But the main takeaway here IMO is the idea about outputting a specific format as controlled by a flag. Of course other tools have done this too in the past. Consider a few Unix utilities reimplemented to work a bit tighter together in terms of how some data is represented, and using for example Cap’n Proto to serialize the data and piping that between them but also writing a library for destructively converting to plain text representation and then using some method or combination of methods for deciding when to use binary piping. For example, - A user-provided flag. Tiresome to type. - A user-provided environment variable. Problematic when mixing tools that implement serialization with pure plaintext tools. - The shell could be aware of which programs implement this serialization and invoke them with the flags or env vars corresponding to whether or not each program on each side of each pipe in a pipeline supports the serialization. Possibly the way to go. In this way one could incrementally rebuild the Unix tools to work this way without having to change everything at once and still being compatible with the universal interface of plaintext forever, since no-one wants nor can implement this in every Unix tool out there. Once you have a couple of tools support this then the possibility for new ways to interact with pipelines becomes available. I still love and will use the classic command-line. But in some situations the possibilities enabled by this would be very compelling indeed.

Longhanks 8y ago
What you describe is PowerShell. It is available for most Linux and macOS.
- codetrotter 8y ago
  I am aware that PowerShell is somewhat like this — in fact I started using PS before permanently switching to Linux back in 2009 — but I don’t enjoy PS. Some people like PS a lot I know and that’s great for them but for me it’s not something I am ever going to use again. I found my home with Unix those 10 years ago.
- pantalaimon 8y ago
  IMHO what kills PowerShell is that it's so incredibly verbose.
  - svnpenn 8y ago
    Even with the built-in aliases?
- int_19h 8y ago
  Not quite. PowerShell input and output are .NET objects - not just data, but also behavior. This requires both sides of the pipe to agree on the object model and the execution environment. JSON (or another pure data serialization format) is much better in that regard, because then you only need to agree on some basic data structures at the boundary.
emmelaich 8y ago
I think csv (with a separator of your choice) would be ideal. It wouldn't require special tools but it could work with them. You could save the column headers, types and separator in extended attributes.
- emmelaich 8y ago
  For extra points you could guarantee the structure of data by only allowing certain tools (selinux context maybe) that are trusted to write the files with the required structure. That would save a lot of repeated wasteful parsing and checking.
caf 8y ago
An alternative that I have had kicking around in the back of my mind for a few years is having the process on the read side of the pipe call fcntl() with a new flag to set an "Accept" set of formats, which the process on the write side of the pipe can retrieve with a 'get' equivalent. If there's a format both sides recognise, they can use that. Both sides just default to plain text so it'll be backwards compatible.
int_19h 8y ago
FreeBSD is using this in its base system (although I don't think it's all or even most apps as yet). http://juniper.github.io/libxo/libxo-manual.html http://juniper.github.io/libxo/libxo-manual.html
chubot 8y ago
I wrote a couple articles about structured data over pipes, which may motivate some features in the Oil shell [1] How to Quickly and Correctly Generate a Git Log in HTML http://www.oilshell.org/blog/2017/09/19.html http://www.oilshell.org/blog/2017/09/19.html Git Log in HTML: A Harder Problem and A Safe Solution http://www.oilshell.org/blog/2017/09/29.html http://www.oilshell.org/blog/2017/09/29.html I'm interested in ideas for structured data over pipes, or even better a prototype and example programs :) [1] http://www.oilshell.org/blog/2018/01/28.html http://www.oilshell.org/blog/2018/01/28.html
wilde 8y ago
This is very similar to PowerShell. But CMD.EXE just kills it for me.