5 ms·

Libxo: Easy way to generate text, XML, JSON, and HTML

89 points by edward 3y ago

ary 3y ago
This, at least in concept, looks like a potential successor to printf() et al. The general accessibility of it is lacking given that it's a C-only API at the moment (there don't appear to be bindings for other languages), and I'm left questioning whether format strings are the best way. Perhaps worse is better in this case. When thinking about this problem I've not been able to get beyond the decision of "should it be done with something like a builder pattern and a graph of objects/structures" or "should it be done with a DSL" (which is what I consider the format strings approach to be). A DSL is more immediately convenient when creating output, but when you want to understand the structure you're emitting it seems better to have code that is explicit and imperative.
- loeg 3y ago
  libxo is in practice a poor approach to generating structured output from unix utilities. There are at least a few problems. The format strings do not easily replace existing formatted prints, so it is not straightforward to adopt. For anything more complicated than simple row records, you have to change the structure of your program significantly and might as well just use a different path for formatting structured output. It is unaware of locales, and as a result, butchers text in non ASCII/UTF-8 encodings. Finally, a separate-binary-with-structured-text-output is a poor library interface to quite a lot of these utilities -- a callable C API would be more broadly useful.
yyyk 3y ago
Libxo is used in FreeBSD. I can't say I'm a fan of the approach though. Typical printf usage is imperative and additive: if (enter) printf("Hello "); else printf("Goodbye "); printf("World!\n"); Using the format string forces the programmer to keep implicit state (the document format) all over the place or get an inconsistent document. For example, imagine the first printf calls the column 'Text' and the others call it 'Output'. We can easily do this for a single format, but the complexity will get higher the more we add. If you do this properly (emit to an object and render from that), the result is trivially consistent. The difficulty here is to get streaming, this is however not always required and can be achieved with a little effort.
- mpweiher 3y ago
  > The difficulty here is to get streaming Polymorphic Write Streams do this. ACM DL: https://dl.acm.org/doi/10.1145/3359619.3359748 https://dl.acm.org/doi/10.1145/3359619.3359748 pdf: http://www.hirschfeld.org/writings/media/WeiherHirschfeld_2019_StandardObjectOutStreamingObjectsWithPolymorphicWriteStreams_AcmDL_Preprint.pdf http://www.hirschfeld.org/writings/media/WeiherHirschfeld_20... Code: https://github.com/mpw/MPWFoundation/tree/master/Streams.subproj https://github.com/mpw/MPWFoundation/tree/master/Streams.sub... Fast JSON parsing using this approach: https://blog.metaobject.com/2020/04/somewhat-less-lethargic-json-support.html https://blog.metaobject.com/2020/04/somewhat-less-lethargic-... Presentation (DLS '19): https://www.youtube.com/watch?v=DG5MtsMojgI https://www.youtube.com/watch?v=DG5MtsMojgI
- lelanthran 3y ago
  > Using the format string forces the programmer to keep implicit state (the document format) all over the place or get an inconsistent document. I'm not understanding your objection[1]; surely you would only define the libxo format string once, and then reuse it everywhere? Without libxo you'd need to duplicate your code everywhere for every output format you want to support. IOW, you'd have to construct your libxo format string using a string concatenation library, something like this: const char *s1 = "{:Text%7ju}"; const char *s2 = "{:Output%7ju}"; char *final = NULL; if (enter) { final = strdup (s1); } else { final = strdup (s2); } final = strconcat (final, s2); Isn't that a better mechanism than printf? [1] It's late, I've the flu and feeling a little stupid right now. Also, this is the first I've seen this project.
  - yyyk 3y ago
    >surely you would only define the libxo format string once, and then reuse it everywhere I would be rather scared of using a variable for a format string. IMHO, these types of format strings are an antipattern for a different reason - we don't see the format at point of use, if we make some too easy mistakes, we have a crash or CVE*. I think nowaways there are tools to do some verification**, and I guess we could use an IDE (but most C programmers don't?), but I am unfamiliar with any such tool which supports libxo style format strings. * https://en.wikipedia.org/wiki/Format_string_attack https://en.wikipedia.org/wiki/Format_string_attack ** IIRC, GCC/Clang eventually added a verifier? But it doesn't apply to all cases or scanf? I don't recall.
JNRowe 3y ago
Every time I see this I'm hoping that it has taken off, because it feels like such an obvious improvement. Instead of that we have some support for it in FreeBSD, and a homegrown solution in some packages like util-linux. Yeah, there are some concerns, but the concept seems sound and the implementation can be iterated upon. For instance, years later and it still isn't packaged in Debian. If nothing out of the tens of thousands of Debian packages has a dependency on it there presumably must be a good reason. It strikes me as one of those libtermkey¹/libvterm things where Leonerd pushed it for years before anybody really used it, despite it being a seemingly obvious improvement over the status quo. ¹ https://www.leonerd.org.uk/code/libtermkey/ https://www.leonerd.org.uk/code/libtermkey/
- ComputerGuru 3y ago
  > Instead of that we have some support for it in FreeBSD My google fu is failing me right now but FreeBSD also has a shared library used for reading/parsing config files and providing either a common or universal dsl for all conf files using the same library. This is one of the benefits of using an OS instead of a distribution - all the tools are developed holistically and refactors such as providing a shared, universal input or output format, sandboxing everything with capsicum, etc across the board are much more possible. EDIT Remembered it. Surprised at how bad Google was at finding this, though! UCL - Universal Configuration Language [0]. Introduced in a paper by Allan Jude in 2015 [1]. Man page: libucl(3) [2]. [0]: https://github.com/vstakhov/libucl/ https://github.com/vstakhov/libucl/ [1]: https://papers.freebsd.org/2015/bsdcan/allanjude-ucl/ https://papers.freebsd.org/2015/bsdcan/allanjude-ucl/ [2]: https://man.freebsd.org/cgi/man.cgi?query=libucl&sektion=3&format=html https://man.freebsd.org/cgi/man.cgi?query=libucl&sektion=3&f...
  - loeg 3y ago
    I wouldn't say ucl or capsicum has seen especially wide adoption inside FreeBSD. The monorepo helps, but the FreeBSD dev community just isn't that big and there isn't a huge incentive to work on using these things.
zokier 3y ago
Seeing that this originates from FreeBSD ecosystem, did it get actually adopted widely in FreeBSD base system? At least I interpreted that to have been the goal: https://juniper.github.io/libxo/libxo-manual.html#can-you-share-the-history-of-libxo https://juniper.github.io/libxo/libxo-manual.html#can-you-sh...
- tedunangst 3y ago
  It's used inconsistently. df uses it, but not du. ps, but not ls.
38 3y ago
you can already do this in other languages. For example, here is Go: package main import ( "encoding/json" "encoding/xml" "os" ) type wc struct { File []file } type file struct { Lines int Words int Characters int Filename string } func main() { etc_motd := wc{ []file{ {25, 1165, 1140, "/etc/motd"}, }, } json.NewEncoder(os.Stdout).Encode(etc_motd) xml.NewEncoder(os.Stdout).Encode(etc_motd) }
- paulddraper 3y ago
  Gee thanks mister!
  - 38 3y ago
    get outta here kid.
kristopolous 3y ago
I bet you could do some pretty clever heuristic hacks to wrap a bunch of programs in this especially if you attach to the process and say, clobber printf. I'm thinking in the spirit of rlwrap. It's certainly more of a game genie approach but it might occasionally be awesome.
Norfair 3y ago
In Haskell we use autodocodec for this.
ComputerGuru 3y ago
Serde can kind of do this for rust projects, but you're usually constrained to outputs that are "identical but for the syntax/format" (i.e. same field names though perhaps with different naming conventions). I've used that to convert configuration files from one language to the other, such as this json2toml and toml2json tool [0]. [0]: https://github.com/neosmart/toml2json https://github.com/neosmart/toml2json
- mananaysiempre 3y ago
  > Serde can kind of do this for rust projects, but you're usually constrained to outputs that are "identical but for the syntax/format" (i.e. same field names though perhaps with different naming conventions). Libxo’s distinguishing feature (IMO) is that its schema specifications are plaintext output with markup specifying which parts are data to be extracted into structured formats, so that you can port your usual Unix tool to it and not ruin its original ad-hoc output. I don’t know of anything positioned as a serialization library that can do this with comparable grace, Serde included. Related: the section on marking up plaintext output in the Ivo essay[1]. [1] https://web.archive.org/web/20111204021526/http://lubutu.com/idea/ivo https://web.archive.org/web/20111204021526/http://lubutu.com... (discussed at the time at https://news.ycombinator.com/item?id=3300264 https://news.ycombinator.com/item?id=3300264)
  - ComputerGuru 3y ago
    Yup. libxo is much more free-form, while anything going through a serialize-deserialize process is going to necessarily have to be more regular.
sam_bristow 3y ago
I would love having structured output from shell commands, but for now I'd settle for people using stdout and stderr correctly.
- moody__ 3y ago
  I see people say they would like this quite often. There is a command line interface that does do this, powershell is very structured in this exact way. But no one is throwing away their sh/bash/tcsh/ksh for powershell. I think this is just a classic example of the grass being more green on the other side.
  - biugbkifcjk 3y ago
    I'd take powershell with proper system integration over bash any day.
    - moody__ 3y ago
      The "with proper system integration" is the kicker. Powershell does run on linux[0], there is nothing stopping you or anyone else in creating this "proper integration", but it has yet to be done. I find this telling in what the real value proposition of this kind of shell is in the first place. I do sympathize with you over bash, I have seen people wield it in quite horrific ways. In general I find that people are under utilizing their functions. For one example, functions in most shell languages have stdin, stdout, and stderr just like programs you call but I often see people miss this and thus miss the ability to compose said functions. I switched over to using the plan9 rc[1] for my scripts years ago and never looked back. [0] https://learn.microsoft.com/en-us/powershell/scripting/install/installing-powershell-on-linux?view=powershell-7.3 https://learn.microsoft.com/en-us/powershell/scripting/insta... [1] http://doc.cat-v.org/plan_9/4th_edition/papers/rc http://doc.cat-v.org/plan_9/4th_edition/papers/rc
      - biugbkifcjk 3y ago
        Yea, I guess I could create that integration, but the key would be getting it included as the default on Linux installs. I like to use what is generally available, so that I'm not completely lost when I remote into an unfamiliar server.

89 points by edward 3y ago

ary 3y ago

This, at least in concept, looks like a potential successor to printf() et al. The general accessibility of it is lacking given that it's a C-only API at the moment (there don't appear to be bindings for other languages), and I'm left questioning whether format strings are the best way. Perhaps worse is better in this case. When thinking about this problem I've not been able to get beyond the decision of "should it be done with something like a builder pattern and a graph of objects/structures" or "should it be done with a DSL" (which is what I consider the format strings approach to be). A DSL is more immediately convenient when creating output, but when you want to understand the structure you're emitting it seems better to have code that is explicit and imperative.

loeg 3y ago
libxo is in practice a poor approach to generating structured output from unix utilities. There are at least a few problems. The format strings do not easily replace existing formatted prints, so it is not straightforward to adopt. For anything more complicated than simple row records, you have to change the structure of your program significantly and might as well just use a different path for formatting structured output. It is unaware of locales, and as a result, butchers text in non ASCII/UTF-8 encodings. Finally, a separate-binary-with-structured-text-output is a poor library interface to quite a lot of these utilities -- a callable C API would be more broadly useful.

yyyk 3y ago

Libxo is used in FreeBSD. I can't say I'm a fan of the approach though. Typical printf usage is imperative and additive: if (enter) printf("Hello "); else printf("Goodbye "); printf("World!\n"); Using the format string forces the programmer to keep implicit state (the document format) all over the place or get an inconsistent document. For example, imagine the first printf calls the column 'Text' and the others call it 'Output'. We can easily do this for a single format, but the complexity will get higher the more we add. If you do this properly (emit to an object and render from that), the result is trivially consistent. The difficulty here is to get streaming, this is however not always required and can be achieved with a little effort.

mpweiher 3y ago
> The difficulty here is to get streaming Polymorphic Write Streams do this. ACM DL: https://dl.acm.org/doi/10.1145/3359619.3359748 https://dl.acm.org/doi/10.1145/3359619.3359748 pdf: http://www.hirschfeld.org/writings/media/WeiherHirschfeld_2019_StandardObjectOutStreamingObjectsWithPolymorphicWriteStreams_AcmDL_Preprint.pdf http://www.hirschfeld.org/writings/media/WeiherHirschfeld_20... Code: https://github.com/mpw/MPWFoundation/tree/master/Streams.subproj https://github.com/mpw/MPWFoundation/tree/master/Streams.sub... Fast JSON parsing using this approach: https://blog.metaobject.com/2020/04/somewhat-less-lethargic-json-support.html https://blog.metaobject.com/2020/04/somewhat-less-lethargic-... Presentation (DLS '19): https://www.youtube.com/watch?v=DG5MtsMojgI https://www.youtube.com/watch?v=DG5MtsMojgI
lelanthran 3y ago
> Using the format string forces the programmer to keep implicit state (the document format) all over the place or get an inconsistent document. I'm not understanding your objection[1]; surely you would only define the libxo format string once, and then reuse it everywhere? Without libxo you'd need to duplicate your code everywhere for every output format you want to support. IOW, you'd have to construct your libxo format string using a string concatenation library, something like this: const char *s1 = "{:Text%7ju}"; const char *s2 = "{:Output%7ju}"; char *final = NULL; if (enter) { final = strdup (s1); } else { final = strdup (s2); } final = strconcat (final, s2); Isn't that a better mechanism than printf? [1] It's late, I've the flu and feeling a little stupid right now. Also, this is the first I've seen this project.
- yyyk 3y ago
  >surely you would only define the libxo format string once, and then reuse it everywhere I would be rather scared of using a variable for a format string. IMHO, these types of format strings are an antipattern for a different reason - we don't see the format at point of use, if we make some too easy mistakes, we have a crash or CVE*. I think nowaways there are tools to do some verification**, and I guess we could use an IDE (but most C programmers don't?), but I am unfamiliar with any such tool which supports libxo style format strings. * https://en.wikipedia.org/wiki/Format_string_attack https://en.wikipedia.org/wiki/Format_string_attack ** IIRC, GCC/Clang eventually added a verifier? But it doesn't apply to all cases or scanf? I don't recall.

JNRowe 3y ago

Every time I see this I'm hoping that it has taken off, because it feels like such an obvious improvement. Instead of that we have some support for it in FreeBSD, and a homegrown solution in some packages like util-linux. Yeah, there are some concerns, but the concept seems sound and the implementation can be iterated upon. For instance, years later and it still isn't packaged in Debian. If nothing out of the tens of thousands of Debian packages has a dependency on it there presumably must be a good reason. It strikes me as one of those libtermkey¹/libvterm things where Leonerd pushed it for years before anybody really used it, despite it being a seemingly obvious improvement over the status quo. ¹ https://www.leonerd.org.uk/code/libtermkey/ https://www.leonerd.org.uk/code/libtermkey/

ComputerGuru 3y ago
> Instead of that we have some support for it in FreeBSD My google fu is failing me right now but FreeBSD also has a shared library used for reading/parsing config files and providing either a common or universal dsl for all conf files using the same library. This is one of the benefits of using an OS instead of a distribution - all the tools are developed holistically and refactors such as providing a shared, universal input or output format, sandboxing everything with capsicum, etc across the board are much more possible. EDIT Remembered it. Surprised at how bad Google was at finding this, though! UCL - Universal Configuration Language [0]. Introduced in a paper by Allan Jude in 2015 [1]. Man page: libucl(3) [2]. [0]: https://github.com/vstakhov/libucl/ https://github.com/vstakhov/libucl/ [1]: https://papers.freebsd.org/2015/bsdcan/allanjude-ucl/ https://papers.freebsd.org/2015/bsdcan/allanjude-ucl/ [2]: https://man.freebsd.org/cgi/man.cgi?query=libucl&sektion=3&format=html https://man.freebsd.org/cgi/man.cgi?query=libucl&sektion=3&f...
- loeg 3y ago
  I wouldn't say ucl or capsicum has seen especially wide adoption inside FreeBSD. The monorepo helps, but the FreeBSD dev community just isn't that big and there isn't a huge incentive to work on using these things.

zokier 3y ago

Seeing that this originates from FreeBSD ecosystem, did it get actually adopted widely in FreeBSD base system? At least I interpreted that to have been the goal: https://juniper.github.io/libxo/libxo-manual.html#can-you-share-the-history-of-libxo https://juniper.github.io/libxo/libxo-manual.html#can-you-sh...

tedunangst 3y ago
It's used inconsistently. df uses it, but not du. ps, but not ls.

38 3y ago

you can already do this in other languages. For example, here is Go: package main import ( "encoding/json" "encoding/xml" "os" ) type wc struct { File []file } type file struct { Lines int Words int Characters int Filename string } func main() { etc_motd := wc{ []file{ {25, 1165, 1140, "/etc/motd"}, }, } json.NewEncoder(os.Stdout).Encode(etc_motd) xml.NewEncoder(os.Stdout).Encode(etc_motd) }

paulddraper 3y ago
Gee thanks mister!
- 38 3y ago
  get outta here kid.

kristopolous 3y ago

I bet you could do some pretty clever heuristic hacks to wrap a bunch of programs in this especially if you attach to the process and say, clobber printf. I'm thinking in the spirit of rlwrap. It's certainly more of a game genie approach but it might occasionally be awesome.

Norfair 3y ago

In Haskell we use autodocodec for this.

ComputerGuru 3y ago

Serde can kind of do this for rust projects, but you're usually constrained to outputs that are "identical but for the syntax/format" (i.e. same field names though perhaps with different naming conventions). I've used that to convert configuration files from one language to the other, such as this json2toml and toml2json tool [0]. [0]: https://github.com/neosmart/toml2json https://github.com/neosmart/toml2json

mananaysiempre 3y ago
> Serde can kind of do this for rust projects, but you're usually constrained to outputs that are "identical but for the syntax/format" (i.e. same field names though perhaps with different naming conventions). Libxo’s distinguishing feature (IMO) is that its schema specifications are plaintext output with markup specifying which parts are data to be extracted into structured formats, so that you can port your usual Unix tool to it and not ruin its original ad-hoc output. I don’t know of anything positioned as a serialization library that can do this with comparable grace, Serde included. Related: the section on marking up plaintext output in the Ivo essay[1]. [1] https://web.archive.org/web/20111204021526/http://lubutu.com/idea/ivo https://web.archive.org/web/20111204021526/http://lubutu.com... (discussed at the time at https://news.ycombinator.com/item?id=3300264 https://news.ycombinator.com/item?id=3300264)
- ComputerGuru 3y ago
  Yup. libxo is much more free-form, while anything going through a serialize-deserialize process is going to necessarily have to be more regular.

sam_bristow 3y ago

I would love having structured output from shell commands, but for now I'd settle for people using stdout and stderr correctly.

moody__ 3y ago
I see people say they would like this quite often. There is a command line interface that does do this, powershell is very structured in this exact way. But no one is throwing away their sh/bash/tcsh/ksh for powershell. I think this is just a classic example of the grass being more green on the other side.
- biugbkifcjk 3y ago
  I'd take powershell with proper system integration over bash any day.
  - moody__ 3y ago
    The "with proper system integration" is the kicker. Powershell does run on linux[0], there is nothing stopping you or anyone else in creating this "proper integration", but it has yet to be done. I find this telling in what the real value proposition of this kind of shell is in the first place. I do sympathize with you over bash, I have seen people wield it in quite horrific ways. In general I find that people are under utilizing their functions. For one example, functions in most shell languages have stdin, stdout, and stderr just like programs you call but I often see people miss this and thus miss the ability to compose said functions. I switched over to using the plan9 rc[1] for my scripts years ago and never looked back. [0] https://learn.microsoft.com/en-us/powershell/scripting/install/installing-powershell-on-linux?view=powershell-7.3 https://learn.microsoft.com/en-us/powershell/scripting/insta... [1] http://doc.cat-v.org/plan_9/4th_edition/papers/rc http://doc.cat-v.org/plan_9/4th_edition/papers/rc
    - biugbkifcjk 3y ago
      Yea, I guess I could create that integration, but the key would be getting it included as the default on Linux installs. I like to use what is generally available, so that I'm not completely lost when I remote into an unfamiliar server.