5 ms·

Show HN: Xq – command-line XML and HTML beautifier and content extractor

104 points by sibprogrammer 4y ago

encryptluks2 4y ago
This is awesome. Hugo has been missing a tidy feature for a long time. I hope they use this to implement it finally or that the author might even consider creating a PR.
samwillis 4y ago
Neat! Like Jq but for XML and HTML. Have you considered adding css sectors as an alternative to xpath? For many simple things a css selector is easer to write and more people already know them. I believe it's possible to translate css selectors to xpath so it wouldn't need another selection engine.
- sibprogrammer 4y ago
  Thank you for your support! I didn't consider CSS selectors as an alternative to Xpath, but really like the idea. Absolutely agree CSS selectors will be more convenient :)
  - esprehn 4y ago
    The issue with CSS selectors is they can only express element selection. There's no concept of picking the text of the element or the value of an attribute. There's been proposals over the years to support pseudo elements for that, so you could make something up like ::attr(name) and ::text, but nothing in the standards. htmlq does this by having various command line options like --attribute and --text. https://github.com/mgdm/htmlq#examples https://github.com/mgdm/htmlq#examples
    - samwillis 4y ago
      Quite a few scraping (such as Scrapy) or browser automated testing frameworks (like Playwright) have standardised on ::text. I would take a look around and if other tools are using the same none standard selectors for these features, it would be perfect for Xq. It's not like you are complying with standards for browser compatibility.
- mk12 4y ago
  htmlq does that.
- deleted 4y ago
  [deleted]
marban 4y ago
What would be the differences to lxml/beautifulsoup/etc.?
kristopolous 4y ago
There's an ancient formatter I've been using for years (gasp, probably well over a decade) https://xml-coreutils.sourceforge.net/ https://xml-coreutils.sourceforge.net/ ... xml-fmt (https://xml-coreutils.sourceforge.net/xml-fmt_man.html https://xml-coreutils.sourceforge.net/xml-fmt_man.html) It'll be nice to try something new. swiss army style command line XML tools has been pretty neglected.
xg15 4y ago
There is also yq [1], which attempts the same for yaml, toml and xml. (And confusingly also contains a binary named "xq" for the xml part - however, it uses jq for querying instead of xpath) [1] https://github.com/kislyuk/yq https://github.com/kislyuk/yq
mikmoila 4y ago
Looks good! Does the xpath expression support XML namespaces?
- mdaniel 4y ago
 No, and arguably worse: it completely ignores them $ echo '<a xmlns="http://www.w3.org/2005/Atom">hello</a>' | xq --xpath '/a/b/text()' hello Only partial support for functions, too, it would appear $ echo '<a xmlns="http://www.w3.org/2005/Atom">hello</a>' | xq --xpath 'count(/a/*)' $ Some A++ error handling, too, as it just calls panic when angry
 - mikmoila 4y ago
 Ah ok; Namespace support before XPath 3.0 with Q{nsname}localname would make cli tool like this a bit too verbose. Of course you could store ns-prefix to ns mappings to conf file or environment variables...
 - mdaniel 4y ago
 I actually tried the lxml syntax "/{urn:beta}b" and that's how I learned about the panic-based error handling, but I didn't know about that Q syntax, so thank you for bringing it to my attention $ echo '<a xmlns="urn:alpha">hello</a>' | xq --xpath '/Q{urn:alpha}a/text()' panic: /Q{urn:alpha}a/text() has an invalid token. goroutine 18 [running]: github.com/antchfx/xmlquery.Find(...) github.com/antchfx/xmlquery@v1.3.8/query.go:76 ... It seems to be a problem with that backing xmlquery library, and I did a quick GitHub topics search and found https://github.com/zzossig/rabbit https://github.com/zzossig/rabbit which claimed to be XPath 3, but then https://github.com/zzossig/rabbit#what-is-not-supported https://github.com/zzossig/rabbit#what-is-not-supported says "lol, namespaces, wat?" so :-(
 - mikmoila 4y ago
 There aren't that many libraries available with full XPath 3.x conformance. In order to give some constructive feedback, perhaps an optional .conf file with ns-prefix->ns-name mappings could be read and passed to the XPath engine as ns-context? Even w/o the XPath functions support and only XPath selector expression this would make the tool to cover most of the usecases eg. simply finding elements.
mdaniel 4y ago
Shoutout to my go-to: https://github.com/EricChiang/pup#readme https://github.com/EricChiang/pup#readme (also golang) and my 2nd favorite https://xmlstar.sourceforge.net/ https://xmlstar.sourceforge.net/
- tmsbrg 4y ago
 Definitely seconding pup, love it and it's easy to use with css selectors. Often use it to parse HTML tables of data on random websites into usable CSVs / etc. xq looks interesting for pure XML, will add it to my notes. Regarding xmlstarlet, I don't use it much but here's some stuff from my notes: XML filter on xpath: xmlstarlet sel -t -c configuration/appender logback.xml; echo XML editing: xmlstarlet ed -d configuration/root -d configuration/logger -s configuration -t elem -n woof -v 'Woof!' logback.xml The last one deletes elements given by xpath "configuration/root" and "configuration/logger", then adds a new element with the contents <woof>Woof!</woof> under the xpath "configuration".
 - mdaniel 4y ago
 > Regarding xmlstarlet, I don't use it much but here's some stuff from my notes: It is also handy doing at some PoC work, and then one can feed it "-C" to output the backing xslt for use elsewhere $ echo '<a>hello</a>' | xmlstarlet sel -t -m /a -c 'count(*)' --nl 1 $ echo '<a>hello</a>' | xmlstarlet sel -C -t -m /a -c 'count(*)' --nl <?xml version="1.0"?> <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0"> <xsl:output omit-xml-declaration="yes" indent="no"/> <xsl:template match="/"> <xsl:for-each select="/a"> <xsl:copy-of select="count(*)"/> <xsl:value-of select="' '"/> </xsl:for-each> </xsl:template> </xsl:stylesheet>
 - tejtm 4y ago
 using `xmlstarlet` to generate XPATHs to visualize document structure https://github.com/TomConlin/xpath2dot https://github.com/TomConlin/xpath2dot
- fbdab103 4y ago
 The pup readme looks pretty great, but a heads up that the project has had limited activity over the past few years. Several raised issues about problems compiling with latest Go releases.
sam_lowry_ 4y ago
xmllint --format - xsltproc
- n8henrie 4y ago
  Yes, I've generally used curl | xmllint --html --xpath for this (usually piped to awk). Not always very pretty, but seems pretty available on MacOS / Linux.
its-summertime 4y ago
doesn't seem to work great with https://ap-playerservices.streamtheworld.com/api/livestream?station=NOVA_919&version=1.9 so xqilla and learning xlst et al still seems like my to-go for complex documents
- manbart 4y ago
  Hadn’t heard of XQilla, but looks like it’s limited to XPath 2. You should try an XPath 3 based implementation. XQuery 3.1 has some nice features not found in XPath 2 implementations e.g. native JSON support, the arrow operator (piping), group by clause in FWLORs. XSLT 3.0 has some nice new stuff too, but I use that less
  - its-summertime 4y ago
    I would switch but, finding a simple binary that does what I want (input -> complex query -> output) is so frustrating. Everything is a framework or library. Thankfully XQilla has its own json support which was all I needed so far
- alecthomas 4y ago
  Works fine for me, what doesn't work? $ curl -fsSL 'https://ap-playerservices.streamtheworld.com/api/livestream?station=NOVA_919&version=1.9' | xq -x //transport http hls hlsts http http hls hlsts http hls hlsts
  - its-summertime 4y ago
    it works if you need to not use the structured nature of the document for $mp in /ls:live_stream_config/ls:mountpoints/ls:mountpoint for $server in $mp/ls:servers/ls:server return { concat( "https://", $server/ls:ip, "/", $mp/ls:mount, ".", local:codec-to-ext($mp/ls:media-format/ls:audio/@codec) ) } is the core of my xqilla work, its something I could replicate in jq, I don't think I could replicate it in xq however
password4321 4y ago
related discussion, orthogonally: JC – JSONifies the output of many CLI tools https://news.ycombinator.com/item?id=33448204 https://news.ycombinator.com/item?id=33448204 2022-11-03
the-printer 4y ago
There is also hred https://github.com/danburzo/hred https://github.com/danburzo/hred Which extracts XML and HTML as JSON. Also cool: https://github.com/dbohdan/structured-text-tools/ https://github.com/dbohdan/structured-text-tools/ "A list of command line tools for manipulating structured text data"
jll29 4y ago
curl https://news.ycombinator.com/ | xq yields: <title>Hacker News</title>XML syntax error on line 4: element <link> closed by </head> 1
- jimsmart 4y ago
 According to the README on Github main page for the project, you need to provide the -m flag when working with HTML. — The tool parses XML by default, which is much stricter than HTML. Hence the error you are seeing (in HTML5, one can omit various close tags and the HTML is still 100% valid — that's what you are encountering here) HTH
nerdponx 4y ago
So there are now approximately 1 million command line tools with various overlapping feature sets for extracting data from XML, JSON, YAML, TOML, and CSV-like delimited data. Is anyone ambitious enough to have constructed a feature matrix for all of them? Is there even a complete list out there? I wonder if this space of tools is a bit like static site generators: it's almost as easy to write your own as it is to learn somebody else's.
abfd 4y ago
[dead]

104 points by sibprogrammer 4y ago

This is awesome. Hugo has been missing a tidy feature for a long time. I hope they use this to implement it finally or that the author might even consider creating a PR.

samwillis 4y ago

Neat! Like Jq but for XML and HTML. Have you considered adding css sectors as an alternative to xpath? For many simple things a css selector is easer to write and more people already know them. I believe it's possible to translate css selectors to xpath so it wouldn't need another selection engine.

sibprogrammer 4y ago
Thank you for your support! I didn't consider CSS selectors as an alternative to Xpath, but really like the idea. Absolutely agree CSS selectors will be more convenient :)
- esprehn 4y ago
  The issue with CSS selectors is they can only express element selection. There's no concept of picking the text of the element or the value of an attribute. There's been proposals over the years to support pseudo elements for that, so you could make something up like ::attr(name) and ::text, but nothing in the standards. htmlq does this by having various command line options like --attribute and --text. https://github.com/mgdm/htmlq#examples https://github.com/mgdm/htmlq#examples
  - samwillis 4y ago
    Quite a few scraping (such as Scrapy) or browser automated testing frameworks (like Playwright) have standardised on ::text. I would take a look around and if other tools are using the same none standard selectors for these features, it would be perfect for Xq. It's not like you are complying with standards for browser compatibility.
mk12 4y ago
htmlq does that.
deleted 4y ago
[deleted]

marban 4y ago

What would be the differences to lxml/beautifulsoup/etc.?

kristopolous 4y ago

There's an ancient formatter I've been using for years (gasp, probably well over a decade) https://xml-coreutils.sourceforge.net/ https://xml-coreutils.sourceforge.net/ ... xml-fmt (https://xml-coreutils.sourceforge.net/xml-fmt_man.html https://xml-coreutils.sourceforge.net/xml-fmt_man.html) It'll be nice to try something new. swiss army style command line XML tools has been pretty neglected.

xg15 4y ago

There is also yq [1], which attempts the same for yaml, toml and xml. (And confusingly also contains a binary named "xq" for the xml part - however, it uses jq for querying instead of xpath) [1] https://github.com/kislyuk/yq https://github.com/kislyuk/yq

mikmoila 4y ago

Looks good! Does the xpath expression support XML namespaces?

mdaniel 4y ago
No, and arguably worse: it completely ignores them $ echo '<a xmlns="http://www.w3.org/2005/Atom">hello</a>' | xq --xpath '/a/b/text()' hello Only partial support for functions, too, it would appear $ echo '<a xmlns="http://www.w3.org/2005/Atom">hello</a>' | xq --xpath 'count(/a/*)' $ Some A++ error handling, too, as it just calls panic when angry
- mikmoila 4y ago
 Ah ok; Namespace support before XPath 3.0 with Q{nsname}localname would make cli tool like this a bit too verbose. Of course you could store ns-prefix to ns mappings to conf file or environment variables...
 - mdaniel 4y ago
 I actually tried the lxml syntax "/{urn:beta}b" and that's how I learned about the panic-based error handling, but I didn't know about that Q syntax, so thank you for bringing it to my attention $ echo '<a xmlns="urn:alpha">hello</a>' | xq --xpath '/Q{urn:alpha}a/text()' panic: /Q{urn:alpha}a/text() has an invalid token. goroutine 18 [running]: github.com/antchfx/xmlquery.Find(...) github.com/antchfx/xmlquery@v1.3.8/query.go:76 ... It seems to be a problem with that backing xmlquery library, and I did a quick GitHub topics search and found https://github.com/zzossig/rabbit https://github.com/zzossig/rabbit which claimed to be XPath 3, but then https://github.com/zzossig/rabbit#what-is-not-supported https://github.com/zzossig/rabbit#what-is-not-supported says "lol, namespaces, wat?" so :-(
 - mikmoila 4y ago
 There aren't that many libraries available with full XPath 3.x conformance. In order to give some constructive feedback, perhaps an optional .conf file with ns-prefix->ns-name mappings could be read and passed to the XPath engine as ns-context? Even w/o the XPath functions support and only XPath selector expression this would make the tool to cover most of the usecases eg. simply finding elements.

mdaniel 4y ago

Shoutout to my go-to: https://github.com/EricChiang/pup#readme https://github.com/EricChiang/pup#readme (also golang) and my 2nd favorite https://xmlstar.sourceforge.net/ https://xmlstar.sourceforge.net/

tmsbrg 4y ago
Definitely seconding pup, love it and it's easy to use with css selectors. Often use it to parse HTML tables of data on random websites into usable CSVs / etc. xq looks interesting for pure XML, will add it to my notes. Regarding xmlstarlet, I don't use it much but here's some stuff from my notes: XML filter on xpath: xmlstarlet sel -t -c configuration/appender logback.xml; echo XML editing: xmlstarlet ed -d configuration/root -d configuration/logger -s configuration -t elem -n woof -v 'Woof!' logback.xml The last one deletes elements given by xpath "configuration/root" and "configuration/logger", then adds a new element with the contents <woof>Woof!</woof> under the xpath "configuration".
- mdaniel 4y ago
 > Regarding xmlstarlet, I don't use it much but here's some stuff from my notes: It is also handy doing at some PoC work, and then one can feed it "-C" to output the backing xslt for use elsewhere $ echo '<a>hello</a>' | xmlstarlet sel -t -m /a -c 'count(*)' --nl 1 $ echo '<a>hello</a>' | xmlstarlet sel -C -t -m /a -c 'count(*)' --nl <?xml version="1.0"?> <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0"> <xsl:output omit-xml-declaration="yes" indent="no"/> <xsl:template match="/"> <xsl:for-each select="/a"> <xsl:copy-of select="count(*)"/> <xsl:value-of select="' '"/> </xsl:for-each> </xsl:template> </xsl:stylesheet>
- tejtm 4y ago
 using `xmlstarlet` to generate XPATHs to visualize document structure https://github.com/TomConlin/xpath2dot https://github.com/TomConlin/xpath2dot
fbdab103 4y ago
The pup readme looks pretty great, but a heads up that the project has had limited activity over the past few years. Several raised issues about problems compiling with latest Go releases.

sam_lowry_ 4y ago

xmllint --format - xsltproc

n8henrie 4y ago
Yes, I've generally used curl | xmllint --html --xpath for this (usually piped to awk). Not always very pretty, but seems pretty available on MacOS / Linux.

its-summertime 4y ago

doesn't seem to work great with https://ap-playerservices.streamtheworld.com/api/livestream?station=NOVA_919&version=1.9 so xqilla and learning xlst et al still seems like my to-go for complex documents

manbart 4y ago
Hadn’t heard of XQilla, but looks like it’s limited to XPath 2. You should try an XPath 3 based implementation. XQuery 3.1 has some nice features not found in XPath 2 implementations e.g. native JSON support, the arrow operator (piping), group by clause in FWLORs. XSLT 3.0 has some nice new stuff too, but I use that less
- its-summertime 4y ago
  I would switch but, finding a simple binary that does what I want (input -> complex query -> output) is so frustrating. Everything is a framework or library. Thankfully XQilla has its own json support which was all I needed so far
alecthomas 4y ago
Works fine for me, what doesn't work? $ curl -fsSL 'https://ap-playerservices.streamtheworld.com/api/livestream?station=NOVA_919&version=1.9' | xq -x //transport http hls hlsts http http hls hlsts http hls hlsts
- its-summertime 4y ago
  it works if you need to not use the structured nature of the document for $mp in /ls:live_stream_config/ls:mountpoints/ls:mountpoint for $server in $mp/ls:servers/ls:server return { concat( "https://", $server/ls:ip, "/", $mp/ls:mount, ".", local:codec-to-ext($mp/ls:media-format/ls:audio/@codec) ) } is the core of my xqilla work, its something I could replicate in jq, I don't think I could replicate it in xq however

password4321 4y ago

related discussion, orthogonally: JC – JSONifies the output of many CLI tools https://news.ycombinator.com/item?id=33448204 https://news.ycombinator.com/item?id=33448204 2022-11-03

the-printer 4y ago

There is also hred https://github.com/danburzo/hred https://github.com/danburzo/hred Which extracts XML and HTML as JSON. Also cool: https://github.com/dbohdan/structured-text-tools/ https://github.com/dbohdan/structured-text-tools/ "A list of command line tools for manipulating structured text data"

jll29 4y ago

curl https://news.ycombinator.com/ | xq yields: <title>Hacker News</title>XML syntax error on line 4: element <link> closed by </head> 1

jimsmart 4y ago
According to the README on Github main page for the project, you need to provide the -m flag when working with HTML. — The tool parses XML by default, which is much stricter than HTML. Hence the error you are seeing (in HTML5, one can omit various close tags and the HTML is still 100% valid — that's what you are encountering here) HTH

nerdponx 4y ago

So there are now approximately 1 million command line tools with various overlapping feature sets for extracting data from XML, JSON, YAML, TOML, and CSV-like delimited data. Is anyone ambitious enough to have constructed a feature matrix for all of them? Is there even a complete list out there? I wonder if this space of tools is a bit like static site generators: it's almost as easy to write your own as it is to learn somebody else's.

abfd 4y ago

[dead]