15 ms·

Htmlq: like jq, but for html

961 points by jabo 5y ago

firefoxd 5y ago
Super useful. You've created a fantastic tool here. Thank you.
Snd_ 5y ago
This is great! Thanks
harperlee 5y ago
Nice! This is the kind of obvious tool that once it exists, you can’t really grok the fact it did not earlier, and that it took until now to exist.
- dmos62 5y ago
  > grok A good opportunity to introduce `gron` to those unfamiliar! ▶ gron "https://api.github.com/repos/tomnomnom/gron/commits?per_page=1" | fgrep "commit.author" json[0].commit.author = {}; json[0].commit.author.date = "2016-07-02T10:51:21Z"; json[0].commit.author.email = "mail@tomnomnom.com"; json[0].commit.author.name = "Tom Hudson"; https://github.com/tomnomnom/gron https://github.com/tomnomnom/gron
  - rsync 5y ago
    "A good opportunity to introduce `gron` to those unfamiliar!" Thank you - appreciated. I haven't done much work with json but have had reasons recently to do so - and I immediately saw how difficult it was to pipeline to grep ... But what I still don't understand is that some json outputs I see have multiple values with the exact same name (!) and that still seems "un-grep-able" to me ... What am I missing ?
    - croon 5y ago
      The json output likely contains multiple objects. Can you request more specifically the object(s) you need and grep on that?
    - dmos62 5y ago
      You might be missing a change in index: `obj[0].prop` vs `obj[1].prop`. Or, your JSON might have the same property defined multiple times: `{a:1, a:2}` (though I'm not sure how gron handles that situation).
      - lvncelot 5y ago
        > (though I'm not sure how gron handles that situation). It seems both gron and jq only use the value that has been defined last: ~ echo '{"a":1,"a":2}' | gron json = {}; json.a = 2; ~ echo '{"a":1,"a":2}' | jq { "a": 2 }
    - dotancohen 5y ago
      > But what I still don't understand is that some json > outputs I see have multiple values with the exact same name This is neither explicitly allowed nor explicitly forbidden by the JSON spec. It is implementation dependent upon how to handle - does one value override the other? Should they be treated as an array? In practice, this situation is usually carefully avoided by services that produce JSON. If you are interfacing with a service that does produce duplicate values, I'd be interested in seeing it for curiosity's sake. If you are writing a service and this is the output, then I implore you to reconsider!
- matsemann 5y ago
  There are already tools for xpath, but using css selectors is much more aligned with what I write every day, so that's nice.
  - harperlee 5y ago
    Yes, and awk and others. I meant something semantically closer to the need, with css selectors.
- ptwt 5y ago
  It did write it a few years ago. https://github.com/plainas/tq https://github.com/plainas/tq
  - mcnesium 5y ago
    and I use it almost every day. It's great, thank you very much!
- natrys 5y ago
  It's not novel obviously. I have been using pup[1] for years. And xidel[2] is probably older. [1] https://github.com/ericchiang/pup https://github.com/ericchiang/pup [2] https://github.com/benibela/xidel https://github.com/benibela/xidel
- mcovey 5y ago
  I have been using hxselect from the html-xml-utils package to do this for many, many years. It doesn't handle malformed HTML that well but can be coaxed into working about 90% of the time, with the help of the other included package hxclean or something like html-tidy.
purplecats 5y ago
brilliant. does this spin up a heavy DOM implementation in the background or do something lighter such as regexp?
- mdzn 5y ago
  You can’t parse HTML with regexps. It’s not a regular language.
  - underdeserver 5y ago
    https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags/1732454#1732454 https://stackoverflow.com/questions/1732348/regex-match-open...
  - carnitine 5y ago
    What language implements regexps that actually correspond to regular languages though?
- Deukhoofd 5y ago
  Looks like it uses servos html5ever (through kuchiki), so no DOM representation.
  - chrismorgan 5y ago
    Kuchiki materialises what they call a “DOM-like tree”. I’d consider it a DOM tree, myself, despite the differences in precise API. But it’s not using a full browser to back it, which I suspect is what’s really being asked.
- mrweasel 5y ago
  It looks to be using html5ever to parse the HTML, similar to something like BeautifulSoup in Python.
- delusional 5y ago
  The source is right there. You can read it. It uses html5ever (part of the servo project).
systemvoltage 5y ago
This is nifty! Python + bs4 takes some googling to remember how to parse a webpage. This is just straight forward, thanks so much.
zatkin 5y ago
Why not incorporate this into jq itself, like perhaps adding some command line arguments to switch to HTML mode?
- Deukhoofd 5y ago
  What would the benefits of fitting a HTML parser into a JSON parser tool be?
  - mjburgess 5y ago
    Well once there's an HTML parser, then a pdf viewer, and then everything needed for PDFs (ie., programming, emailer, video support, etc.) we'll finally have that ideal operating system we've been waiting for.
  - lmm 5y ago
    JQ is not just a parser but a tool for doing operations, many of which are (or should be) generic across any tree-like data format. Reusing that part across different input formats makes a lot of sense.
  - mro_name 5y ago
    sounds a lot more like blockchain.
- e12e 5y ago
  Would probably be more useful to implement html2json, and pipe in html? Ed: eg: https://github.com/Jxck/html2json https://github.com/Jxck/html2json
deleted 5y ago
[deleted]
gizdan 5y ago
Once upon a time I was using pup[0] for such thing as well as later I changed to cascadia[1] which seemed much more advanced. Comparing the two repos, it seems pup is dead, but cascadia may not be. These tools, including htmlq, seem to sell themselves as "jq for html", which is far from the truth. Jq is closer to the awk where you can do just about everything with json. Cascadia, htmlq, and pup seem closer to grep for html. They can essentially only select data from a html source. [0] https://github.com/EricChiang/pup https://github.com/EricChiang/pup [1] https://github.com/suntong/cascadia https://github.com/suntong/cascadia
- croon 5y ago
  Well, jq is grep as well as sed and awk, but yeah, htmlq seems to be just grep, for sake of comparison. But I don't think html has any need for a sed/awk tool, or at least not as much. Json output could very well be piped forward to the next CLI tool after you've changed it slightly with jq. I don't see this scenario as likely with html.
  - gizdan 5y ago
    > Well, jq is grep as well as sed and awk, but yeah, htmlq seems to be just grep, for sake of comparison. Exactly, and that is what I mean. If you want to compare, compare it with grep, not jq. Someone else posted xidel[0] in this thread, which I've not used, but it seems to be the "jq but for html". [0] https://github.com/benibela/xidel https://github.com/benibela/xidel
- heavyset_go 5y ago
  I've used pup for a few projects, but was unaware of cascadia. Thanks for pointing it out.
andybak 5y ago
> like jg "jq is a lightweight and flexible command-line JSON processor"
avereveard 5y ago
what's wrong with using html tidy + xmllint ?
- mro_name 5y ago
  nothing wrong. Searching unmodified html though is sometimes preferable.
triska 5y ago
This is very nice! For reasoning about tree-based data such as HTML, I also highly recommend the declarative programming language Prolog. HTML documents map naturally to Prolog terms and can be readily reasoned about with built-in language mechanisms. For instance, here is the sample query from the htmlq README, fetching all elements with id get-help from https://www.rust-lang.org https://www.rust-lang.org, using Scryer Prolog and its SGML and HTTP libraries in combination with the XPath-inspired query language from library(xpath): ?- http_open("https://www.rust-lang.org", Stream, []), load_html(stream(Stream), DOM, []), xpath(DOM, //(*(@id="get-help")), E). Yielding: E = element(div,[class="flex flex-colum ...",id="get-help"],["\n ",element(h4,[],["Get help!"]),"\n ",element(ul,[],["\n ...",element(li,[],[element(a,[... = ...],[...])]),"\n ...",element(li,[],[...]),...|...]),"\n ...",element(div,[class="la ..."],["\n ...",element(label,[...],[...]),...|...]),"\n ..."]) ; false. The selector //(*(@id="get-help")) is used to obtain all HTML elements whose id attribute is get-help. On backtracking, all solutions are reported. The other example from the README, extracting all links from the page, can be obtained with Scryer Prolog like this: ?- http_open("https://www.rust-lang.org", Stream, []), load_html(stream(Stream), DOM, []), xpath(DOM, //a(@href), Link), portray_clause(Link), false. This query uses forced backtracking to write all links on standard output, yielding: "/". "/tools/install". "/learn". "https://play.rust-lang.org/". "/tools". "/governance". "/community". "https://blog.rust-lang.org/". "/learn/get-started". etc.
- chriswarbo 5y ago
  Thanks, that's a rare example of something which is (a) simple enough to understand for a Prolog-newbie like me, and (b) more practical than ubiquitous family-tree example. I'm always looking for opportunities to dip my toes into Prolog; in hindsight it's clearly a good fit for tree-structured data structures.
  - samhw 5y ago
    Interestingly, the only other context in which I've come across Prolog is from friends who studied at Cambridge, here in the UK. For some reason, the CS 'tripos' (course) there is really heavily focussed on Prolog, and everyone I know from there ended up a huge fan of the language. I'm not sure why that's the case, though, given that almost all other universities seem to use more common languages (Java, C++, etc).
dfederschmidt 5y ago
This looks very useful, big fan of all the ^[a-z]+q$ utilities out there. But as a user, I would probably want to use XPath[0] notation here. Maybe that is just me. A quick search revealed xidel[1] which seems to be similar, but supports XPath. [0]https://en.wikipedia.org/wiki/XPath https://en.wikipedia.org/wiki/XPath [1]https://github.com/benibela/xidel https://github.com/benibela/xidel
- chriswarbo 5y ago
  My web scraping tends to start with xidel. If I need a little bit more power I'll use xmlstarlet. If neither of those is enough, I'll use Python's beautifulsoup package :)
  - mikepurvis 5y ago
    I like xmlstarlet too, if only because it's old enough that I can reliably get it in package repositories and the dependency footprint is tiny (less an issue now with this tool written in Rust, but previously I was comparing to NPM- and PyPI-based affairs).
  - spiralx 5y ago
    lxml is one of the most pleasing to use Python libraries ever, managing to wrap a hot mess of XML APIs in a consistent and Pythonic fashion that you rarely need to escape. IIRC I used beautifulsoup to parse the HTML of a site, and then lxml and either find items and fields by CSS in IPython for quick and dirty data munging, or knock up an XSLT file to transform what I'd scraped into good data in an XML file :)
- akie 5y ago
  I'd like to state my support for the author's choice of CSS selectors in this particular use case. I think it's a natural fit for this domain and already very well known, perhaps even known better than XPath.
  - berkes 5y ago
    I'd like to add my support here too, but with a note. When scraping and parsing (or writing integration test DSL), I always start out with CSS selectors. But always hit cases where they lack or require hoop-jumping and then fall back on Xpath. I then have a codebase with both CSS-Sel and Xpath, which is arguably worse then having only one method. I suspect here, one uses this tool untill CSS selector limitations are getting in the way, after which one switches to another tool(chain)
dredmorbius 5y ago
See also the html-xml-utils from w3c. hxextract and hxselect perform similar extract functions. hxclean and hxnormalize (combined) will pretty-print HTML. https://www.w3.org/Tools/HTML-XML-utils/ https://www.w3.org/Tools/HTML-XML-utils/
- mozey 5y ago
  Funny, couple of years ago I thought someone should create something for JSON similar to what [XSLT](https://en.wikipedia.org/wiki/XSLT https://en.wikipedia.org/wiki/XSLT) is for XML. See example here https://www.w3schools.com/xml/xsl_intro.asp https://www.w3schools.com/xml/xsl_intro.asp Then I found out about jq because awscli was using it in example docs. I guess `htmlq` makes sense if it has the exact same syntax as `jq`, and the user is already familiar with the latter?
  - spiralx 5y ago
    JSON schemas are a thing that exists and can be useful, and `jq` probably covers the 80% of use cases for querying JSON. XSLT can be an amazing tool when used properly and I've wondered about a JS equivalent over the years and started writing one on a couple of occasions. But JSON is just a data structure and not structured markup, and there's no sweet spot for a transformation tool like XSLT - you're more likely to be doing a "find items in JSON, filter() them, then map()/reduce() to output format" task that takes a minute or two in Node and then never gets used again, or doing a complete map from one domain to another where you'd need to do it in JS because of the complexity of processing and ability to handle errors, use third-party tools and even write tests. An XQuery-esque language allowing selecting bits of JSON file(s) with filtering, grouping and ordering built-in, combined with a way of projecting results that's no worse than JS allows for i.e. not having to put quotes around everything and the like :)
gigatexal 5y ago
This is very cool. This will make scraping the web even easier!
m4r35n357 5y ago
Should be HQ . . .
pkrumins 5y ago
Call it "hq".
hyperpallium2 5y ago
From examples, this is only like jq in the sense that the q stands for the same thing. Even the way it does that is different. An xmlq that was really like jq would be fun, about 20 years ago.
- cerved 5y ago
  I would still like xmlq, there are (regrettably) still a lot of applications that store data and configuration in xml
- dotancohen 5y ago
  There is `xq` today, which parses XML like `jq`. I think that it is relatively unknown because it is part of the `yq` package for parsing YMAL. So just install `yq` via PIP and you'll get `xq` as well. There is also `xmlstarlet` for parsing XML in a similar fashion.
  - jle17 5y ago
    Just looked into this and I think it's worth mentioning that there are two different projects called `yq`. The first one that came up (written in go instead of python) is not the right one and doesn't have the `xq` tool.
  - hyperpallium2 5y ago
    xmlstarlet is really nothing like jq, as a language. But yes, I use it because it is the best commandline xml processor I'd found. That's the only similarity to jq. Is this the yq? https://kislyuk.github.io/yq/ https://kislyuk.github.io/yq/ It does contain an 'xq', as a literal wrapper for jq, piping output into it after transcoding XML to JSON using xmltodict https://github.com/martinblech/xmltodict https://github.com/martinblech/xmltodict (which explodes xml into separate JSON data structures). This is a bash one-liner! But TBF it really is a 'jq for xml'. I think it would be horrible for some things, but you could also do a lot of useful things painlessly.
    - dotancohen 5y ago
ducktective 5y ago
Looks nice! Any comparisons with pup? https://github.com/ericchiang/pup https://github.com/ericchiang/pup
unityByFreedom 5y ago
Why not just jquery?
jillesvangurp 5y ago
If you make the html well formed, xpath also works great. Great stuff if you ever need to pick html apart. Used this quite a bit when microformats were still a thing together with jtidy. Jq is very loosely inspired by that, I guess. Might come full circle here and use some XSL transformations ...
- qw 5y ago
  You can usually find a html parser for your language, that you can use xpath/xsl on. It will just make the same assumptions that the browser does, by adding missing closing tags etc. I made a tool that extracted parts of web pages 10-15 years ago, and it worked well. There are of course cases where the html is so unstructured that the results were unpredictable, but it worked well in general.
pabs3 5y ago
I tend to reach for XPath selectors before CSS ones when querying HTML.
mro_name 5y ago
it's statically linkable rust, isn't it? Awesome. I'm looking for a successor to $ xmllint --html --xpath … that doesn't choke on inline svg.
teitoklien 5y ago
Maybe call it hq ?
- Simplicitas 5y ago
  My thoughts EXACTLY... but anyway, great new utility indeed!
  - teitoklien 5y ago
    Haha, Indeed its a very good utility :D
srg0 5y ago
"htmlq: like jq, but for HTML" "jq is like sed for JSON data" sed: "While in some ways similar to an editor which permits scripted edits (_such as ed_), sed works by making only one pass over the input(s)" ed: "ed is a line-oriented text editor". Software definition through a reference to another software is somewhat confusing. Potential users come from different backgrounds (I had no idea what is jq), and it is not clear what are the defining features of each project. Is jq line oriented? Is htmlq operating in a single pass?
- oauea 5y ago
  What's this thing called a "computer" that people keep going on about, anyway?
  - theandrewbailey 5y ago
    It's like a programmable loom, but for logical and mathematical operations.
    - samhw 5y ago
      You may be interested in the symbol grounding problem (https://en.wikipedia.org/wiki/Symbol_grounding_problem#Grounding_process https://en.wikipedia.org/wiki/Symbol_grounding_problem#Groun...). It's like the binding problem, but for symbols.
      - deleted 5y ago
        [deleted]
  - da_chicken 5y ago
    It's a person who does mathematic calculations all day. For example, creating range tables for artillery, calculating averages or totals of a large range of values, or solving complex integrals or differential equations, and so on. They're commonly used in industry or government, especially in astronomy, aerospace and civil engineering for both simulation and analysis. Perhaps the most well-known computers were the Harvard Computers, which operated in the late 19th and early 20th centuries. As a job, computers were largely automated out of existence by solid-state transistor based automated computers and integrated circuit transistor automated computers in the 60s, 70s and 80s, which replaced the enormously expensive and often largely experimental electro-mechanical automated computers while radically reducing cost and improving performance both by several orders of magnitude.
abledon 5y ago
is anyone else using the https://github.com/json-path/JsonPath https://github.com/json-path/JsonPath over the jq route? I hope we standardize on some jq query language, like we have with a base set of SQL syntax
d--b 5y ago
Just being that guy: is there a reason you didn't call it hq?
- zamadatix 5y ago
  Not author but neither is the poster: Jq got away with it because it's one of the few 2 letter combinations that wasn't absolutely overloaded and "jquery" was already taken. OTOH nobody shortens HTML to H and HQ is an extremely common acronym, if not one of the most popular 2 letter acronyms you could pick.
  - OJFord 5y ago
    jq didn't get away with it! Have you never tried searching for anything to do with it? How I wish it were called `jsonq`!
- mgdm 5y ago
  I just wanted to be slightly more descriptive and less likely to collide with other tools.
  - notRobot 5y ago
    Hahah, I love how this is your second comment in 10 years on HN.
    - mgdm 5y ago
      Hah. Yeah. I had another account for a little while but then HN started to let me reset the password for this one quite recently, so here I am.
parhamn 5y ago
Ive been looking for a library that can find the best set of selectors to most consistently find the element youre looking for in a page. Any pointers to something that exists? Interestingly I've also found very little for dom extraction in the OS ML space.
Ronak123 5y ago
https://techflashes.com/top-upcoming-futuristic-technologies/ https://techflashes.com/top-upcoming-futuristic-technologies...
desktopninja 5y ago
Very nice tool. I've long spoiled myself with Powershell's: Invoke-WebRequest eg. # what is the latest release of apache-tomcat? $LINKS=$(Invoke-WebRequest -Uri 'https://tomcat.apache.org/download-80.cgi' | Select-Object -ExpandProperty Links) $LATEST=$($Links | Where-Object -Property href -Match '#8.5.[0-9]+').href.substring(1) $FETCH=$($Links | Where-Object -Property href -match "apache-tomcat-${LATEST}.zip$").href
- Tepix 5y ago
  Should it be $LINKS instead of $Links (2x)?
  - desktopninja 5y ago
    "$links" works too because PWSH is not case sensitive. But I should have used $LINKS like you said for cleaner write-up.
necovek 5y ago
Nice, I expected something based on XPath (like xpd), but web developers dealing with HTML are infinitely more familiar with CSS selectors, so a great choice!
- busterarm 5y ago
  I want the option to use both, like Nokogiri gives you.
  - necovek 5y ago
    Sure, that sounds nice, but having two simple tools each doing the job well in its own space is perfectly fine for me — do you imagine needing to combine Xpath and CSS queries in a single run?
    - busterarm 5y ago
      I've had to do it when dealing with some poorly-designed XML apis in the past. Nokogiri was a godsend.
who-shot-jr 5y ago
Good work!
oauea 5y ago
https://jsoup.org/ https://jsoup.org/ has been around for a long time and seems a bit more mature & maintained than this two-code-files 2-year-old repo. Highly recommend.
mcovalt 5y ago
I’d like to see a tool using lol-html [0] and their CSS selector API as a streaming HTML editor. [0] https://github.com/cloudflare/lol-html https://github.com/cloudflare/lol-html
chefandy 5y ago
If anyone is looking for a good library to do this in Python, PyQuery works well: https://pythonhosted.org/pyquery/ https://pythonhosted.org/pyquery/
elif 5y ago
When I saw the title I thought this was some machine learning-specific rmq/0mq message passing tech called HT. Very excited to zero.
soheil 5y ago
I'd use something like this script that you can put together yourself: #!/usr/bin/env ruby require 'nokogiri'; p Nokogiri::HTML(STDIN.read).css(ARGV[0]).text Just save it to a file in your /usr/local/bin/hq and do chmod +x !$ Then you can do: curl -s "https://news.ycombinator.com/news"|hq "tr:first-child .storylink" It uses Nokogiri[0], which is much more battle tested and works with CSS and XPath selectors. [0] https://nokogiri.org/tutorials/parsing_an_html_xml_document.html#from-a-string https://nokogiri.org/tutorials/parsing_an_html_xml_document....
- drran 5y ago
  Command just prints a bunch of text stitched together: curl -s "https://news.ycombinator.com/news"|hq "tr .storylink" Deploy a website on imgur.comMy £4 a month server can handle 4.2M requests a dayFirst Edition...
  - soheil 5y ago
    Make sure you add :first-child as in my example, otherwise you'll get all the stories smooshed together.
    - drran 5y ago
      I want to see all stories, not first one only.
notorandit 5y ago
Next is xmlq: https://github.com/dscape/xmlq https://github.com/dscape/xmlq
downWidOutaFite 5y ago
Why? I find xpath's syntax much simpler and regular than jq's.
ludovicianul 5y ago
And a Java version with pre-compiled binaries: https://github.com/ludovicianul/hq https://github.com/ludovicianul/hq
bamdadd 5y ago
is there a brew install command ?
rendall 5y ago
What is jq?
jhatemyjob 5y ago
Crazy how a 300-line codebase manages to amass 2000 stars on Github and 700 upvotes on HN. Amazing ROI.
gravypod 5y ago
This looks really cool! I'd love to see a generic query language/tool library for structured data.
bandie91 5y ago
shameless self promotion: parsel[0] is a python script in front of the identically named python lib, and extracts parts of the HTML by CSS selector. the advantage of it compared to most similar tools is that you can navigate in the DOM tree up and down to find precisely what you want if the HTML is poorly marked up, or the searched parts are not close to each other. [0] https://github.com/bAndie91/tools/blob/master/usr/bin/parsel https://github.com/bAndie91/tools/blob/master/usr/bin/parsel

961 points by jabo 5y ago

firefoxd 5y ago

Super useful. You've created a fantastic tool here. Thank you.

Snd_ 5y ago

This is great! Thanks

harperlee 5y ago

Nice! This is the kind of obvious tool that once it exists, you can’t really grok the fact it did not earlier, and that it took until now to exist.

dmos62 5y ago
> grok A good opportunity to introduce `gron` to those unfamiliar! ▶ gron "https://api.github.com/repos/tomnomnom/gron/commits?per_page=1" | fgrep "commit.author" json[0].commit.author = {}; json[0].commit.author.date = "2016-07-02T10:51:21Z"; json[0].commit.author.email = "mail@tomnomnom.com"; json[0].commit.author.name = "Tom Hudson"; https://github.com/tomnomnom/gron https://github.com/tomnomnom/gron
- rsync 5y ago
  "A good opportunity to introduce `gron` to those unfamiliar!" Thank you - appreciated. I haven't done much work with json but have had reasons recently to do so - and I immediately saw how difficult it was to pipeline to grep ... But what I still don't understand is that some json outputs I see have multiple values with the exact same name (!) and that still seems "un-grep-able" to me ... What am I missing ?
  - croon 5y ago
    The json output likely contains multiple objects. Can you request more specifically the object(s) you need and grep on that?
  - dmos62 5y ago
    You might be missing a change in index: `obj[0].prop` vs `obj[1].prop`. Or, your JSON might have the same property defined multiple times: `{a:1, a:2}` (though I'm not sure how gron handles that situation).
    - lvncelot 5y ago
      > (though I'm not sure how gron handles that situation). It seems both gron and jq only use the value that has been defined last: ~ echo '{"a":1,"a":2}' | gron json = {}; json.a = 2; ~ echo '{"a":1,"a":2}' | jq { "a": 2 }
  - dotancohen 5y ago
    > But what I still don't understand is that some json > outputs I see have multiple values with the exact same name This is neither explicitly allowed nor explicitly forbidden by the JSON spec. It is implementation dependent upon how to handle - does one value override the other? Should they be treated as an array? In practice, this situation is usually carefully avoided by services that produce JSON. If you are interfacing with a service that does produce duplicate values, I'd be interested in seeing it for curiosity's sake. If you are writing a service and this is the output, then I implore you to reconsider!
matsemann 5y ago
There are already tools for xpath, but using css selectors is much more aligned with what I write every day, so that's nice.
- harperlee 5y ago
  Yes, and awk and others. I meant something semantically closer to the need, with css selectors.
ptwt 5y ago
It did write it a few years ago. https://github.com/plainas/tq https://github.com/plainas/tq
- mcnesium 5y ago
  and I use it almost every day. It's great, thank you very much!
natrys 5y ago
It's not novel obviously. I have been using pup[1] for years. And xidel[2] is probably older. [1] https://github.com/ericchiang/pup https://github.com/ericchiang/pup [2] https://github.com/benibela/xidel https://github.com/benibela/xidel
mcovey 5y ago
I have been using hxselect from the html-xml-utils package to do this for many, many years. It doesn't handle malformed HTML that well but can be coaxed into working about 90% of the time, with the help of the other included package hxclean or something like html-tidy.

purplecats 5y ago

brilliant. does this spin up a heavy DOM implementation in the background or do something lighter such as regexp?

mdzn 5y ago
You can’t parse HTML with regexps. It’s not a regular language.
- underdeserver 5y ago
  https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags/1732454#1732454 https://stackoverflow.com/questions/1732348/regex-match-open...
- carnitine 5y ago
  What language implements regexps that actually correspond to regular languages though?
Deukhoofd 5y ago
Looks like it uses servos html5ever (through kuchiki), so no DOM representation.
- chrismorgan 5y ago
  Kuchiki materialises what they call a “DOM-like tree”. I’d consider it a DOM tree, myself, despite the differences in precise API. But it’s not using a full browser to back it, which I suspect is what’s really being asked.
mrweasel 5y ago
It looks to be using html5ever to parse the HTML, similar to something like BeautifulSoup in Python.
delusional 5y ago
The source is right there. You can read it. It uses html5ever (part of the servo project).

systemvoltage 5y ago

This is nifty! Python + bs4 takes some googling to remember how to parse a webpage. This is just straight forward, thanks so much.

zatkin 5y ago

Why not incorporate this into jq itself, like perhaps adding some command line arguments to switch to HTML mode?

Deukhoofd 5y ago
What would the benefits of fitting a HTML parser into a JSON parser tool be?
- mjburgess 5y ago
  Well once there's an HTML parser, then a pdf viewer, and then everything needed for PDFs (ie., programming, emailer, video support, etc.) we'll finally have that ideal operating system we've been waiting for.
- lmm 5y ago
  JQ is not just a parser but a tool for doing operations, many of which are (or should be) generic across any tree-like data format. Reusing that part across different input formats makes a lot of sense.
- mro_name 5y ago
  sounds a lot more like blockchain.
e12e 5y ago
Would probably be more useful to implement html2json, and pipe in html? Ed: eg: https://github.com/Jxck/html2json https://github.com/Jxck/html2json

deleted 5y ago

[deleted]

gizdan 5y ago

Once upon a time I was using pup[0] for such thing as well as later I changed to cascadia[1] which seemed much more advanced. Comparing the two repos, it seems pup is dead, but cascadia may not be. These tools, including htmlq, seem to sell themselves as "jq for html", which is far from the truth. Jq is closer to the awk where you can do just about everything with json. Cascadia, htmlq, and pup seem closer to grep for html. They can essentially only select data from a html source. [0] https://github.com/EricChiang/pup https://github.com/EricChiang/pup [1] https://github.com/suntong/cascadia https://github.com/suntong/cascadia

croon 5y ago
Well, jq is grep as well as sed and awk, but yeah, htmlq seems to be just grep, for sake of comparison. But I don't think html has any need for a sed/awk tool, or at least not as much. Json output could very well be piped forward to the next CLI tool after you've changed it slightly with jq. I don't see this scenario as likely with html.
- gizdan 5y ago
  > Well, jq is grep as well as sed and awk, but yeah, htmlq seems to be just grep, for sake of comparison. Exactly, and that is what I mean. If you want to compare, compare it with grep, not jq. Someone else posted xidel[0] in this thread, which I've not used, but it seems to be the "jq but for html". [0] https://github.com/benibela/xidel https://github.com/benibela/xidel
heavyset_go 5y ago
I've used pup for a few projects, but was unaware of cascadia. Thanks for pointing it out.

andybak 5y ago

> like jg "jq is a lightweight and flexible command-line JSON processor"

avereveard 5y ago

what's wrong with using html tidy + xmllint ?

mro_name 5y ago
nothing wrong. Searching unmodified html though is sometimes preferable.

triska 5y ago

This is very nice! For reasoning about tree-based data such as HTML, I also highly recommend the declarative programming language Prolog. HTML documents map naturally to Prolog terms and can be readily reasoned about with built-in language mechanisms. For instance, here is the sample query from the htmlq README, fetching all elements with id get-help from https://www.rust-lang.org https://www.rust-lang.org, using Scryer Prolog and its SGML and HTTP libraries in combination with the XPath-inspired query language from library(xpath): ?- http_open("https://www.rust-lang.org", Stream, []), load_html(stream(Stream), DOM, []), xpath(DOM, //(*(@id="get-help")), E). Yielding: E = element(div,[class="flex flex-colum ...",id="get-help"],["\n ",element(h4,[],["Get help!"]),"\n ",element(ul,[],["\n ...",element(li,[],[element(a,[... = ...],[...])]),"\n ...",element(li,[],[...]),...|...]),"\n ...",element(div,[class="la ..."],["\n ...",element(label,[...],[...]),...|...]),"\n ..."]) ; false. The selector //(*(@id="get-help")) is used to obtain all HTML elements whose id attribute is get-help. On backtracking, all solutions are reported. The other example from the README, extracting all links from the page, can be obtained with Scryer Prolog like this: ?- http_open("https://www.rust-lang.org", Stream, []), load_html(stream(Stream), DOM, []), xpath(DOM, //a(@href), Link), portray_clause(Link), false. This query uses forced backtracking to write all links on standard output, yielding: "/". "/tools/install". "/learn". "https://play.rust-lang.org/". "/tools". "/governance". "/community". "https://blog.rust-lang.org/". "/learn/get-started". etc.

chriswarbo 5y ago
Thanks, that's a rare example of something which is (a) simple enough to understand for a Prolog-newbie like me, and (b) more practical than ubiquitous family-tree example. I'm always looking for opportunities to dip my toes into Prolog; in hindsight it's clearly a good fit for tree-structured data structures.
- samhw 5y ago
  Interestingly, the only other context in which I've come across Prolog is from friends who studied at Cambridge, here in the UK. For some reason, the CS 'tripos' (course) there is really heavily focussed on Prolog, and everyone I know from there ended up a huge fan of the language. I'm not sure why that's the case, though, given that almost all other universities seem to use more common languages (Java, C++, etc).

dfederschmidt 5y ago

This looks very useful, big fan of all the ^[a-z]+q$ utilities out there. But as a user, I would probably want to use XPath[0] notation here. Maybe that is just me. A quick search revealed xidel[1] which seems to be similar, but supports XPath. [0]https://en.wikipedia.org/wiki/XPath https://en.wikipedia.org/wiki/XPath [1]https://github.com/benibela/xidel https://github.com/benibela/xidel

chriswarbo 5y ago
My web scraping tends to start with xidel. If I need a little bit more power I'll use xmlstarlet. If neither of those is enough, I'll use Python's beautifulsoup package :)
- mikepurvis 5y ago
  I like xmlstarlet too, if only because it's old enough that I can reliably get it in package repositories and the dependency footprint is tiny (less an issue now with this tool written in Rust, but previously I was comparing to NPM- and PyPI-based affairs).
- spiralx 5y ago
  lxml is one of the most pleasing to use Python libraries ever, managing to wrap a hot mess of XML APIs in a consistent and Pythonic fashion that you rarely need to escape. IIRC I used beautifulsoup to parse the HTML of a site, and then lxml and either find items and fields by CSS in IPython for quick and dirty data munging, or knock up an XSLT file to transform what I'd scraped into good data in an XML file :)
akie 5y ago
I'd like to state my support for the author's choice of CSS selectors in this particular use case. I think it's a natural fit for this domain and already very well known, perhaps even known better than XPath.
- berkes 5y ago
  I'd like to add my support here too, but with a note. When scraping and parsing (or writing integration test DSL), I always start out with CSS selectors. But always hit cases where they lack or require hoop-jumping and then fall back on Xpath. I then have a codebase with both CSS-Sel and Xpath, which is arguably worse then having only one method. I suspect here, one uses this tool untill CSS selector limitations are getting in the way, after which one switches to another tool(chain)

dredmorbius 5y ago

See also the html-xml-utils from w3c. hxextract and hxselect perform similar extract functions. hxclean and hxnormalize (combined) will pretty-print HTML. https://www.w3.org/Tools/HTML-XML-utils/ https://www.w3.org/Tools/HTML-XML-utils/

mozey 5y ago
Funny, couple of years ago I thought someone should create something for JSON similar to what [XSLT](https://en.wikipedia.org/wiki/XSLT https://en.wikipedia.org/wiki/XSLT) is for XML. See example here https://www.w3schools.com/xml/xsl_intro.asp https://www.w3schools.com/xml/xsl_intro.asp Then I found out about jq because awscli was using it in example docs. I guess `htmlq` makes sense if it has the exact same syntax as `jq`, and the user is already familiar with the latter?
- spiralx 5y ago
  JSON schemas are a thing that exists and can be useful, and `jq` probably covers the 80% of use cases for querying JSON. XSLT can be an amazing tool when used properly and I've wondered about a JS equivalent over the years and started writing one on a couple of occasions. But JSON is just a data structure and not structured markup, and there's no sweet spot for a transformation tool like XSLT - you're more likely to be doing a "find items in JSON, filter() them, then map()/reduce() to output format" task that takes a minute or two in Node and then never gets used again, or doing a complete map from one domain to another where you'd need to do it in JS because of the complexity of processing and ability to handle errors, use third-party tools and even write tests. An XQuery-esque language allowing selecting bits of JSON file(s) with filtering, grouping and ordering built-in, combined with a way of projecting results that's no worse than JS allows for i.e. not having to put quotes around everything and the like :)

gigatexal 5y ago

This is very cool. This will make scraping the web even easier!

m4r35n357 5y ago

Should be HQ . . .

pkrumins 5y ago

Call it "hq".

hyperpallium2 5y ago

From examples, this is only like jq in the sense that the q stands for the same thing. Even the way it does that is different. An xmlq that was really like jq would be fun, about 20 years ago.

cerved 5y ago
I would still like xmlq, there are (regrettably) still a lot of applications that store data and configuration in xml
dotancohen 5y ago
There is `xq` today, which parses XML like `jq`. I think that it is relatively unknown because it is part of the `yq` package for parsing YMAL. So just install `yq` via PIP and you'll get `xq` as well. There is also `xmlstarlet` for parsing XML in a similar fashion.
- jle17 5y ago
  Just looked into this and I think it's worth mentioning that there are two different projects called `yq`. The first one that came up (written in go instead of python) is not the right one and doesn't have the `xq` tool.
- hyperpallium2 5y ago
  xmlstarlet is really nothing like jq, as a language. But yes, I use it because it is the best commandline xml processor I'd found. That's the only similarity to jq. Is this the yq? https://kislyuk.github.io/yq/ https://kislyuk.github.io/yq/ It does contain an 'xq', as a literal wrapper for jq, piping output into it after transcoding XML to JSON using xmltodict https://github.com/martinblech/xmltodict https://github.com/martinblech/xmltodict (which explodes xml into separate JSON data structures). This is a bash one-liner! But TBF it really is a 'jq for xml'. I think it would be horrible for some things, but you could also do a lot of useful things painlessly.
  - dotancohen 5y ago

ducktective 5y ago

Looks nice! Any comparisons with pup? https://github.com/ericchiang/pup https://github.com/ericchiang/pup

unityByFreedom 5y ago

Why not just jquery?

jillesvangurp 5y ago

If you make the html well formed, xpath also works great. Great stuff if you ever need to pick html apart. Used this quite a bit when microformats were still a thing together with jtidy. Jq is very loosely inspired by that, I guess. Might come full circle here and use some XSL transformations ...

qw 5y ago
You can usually find a html parser for your language, that you can use xpath/xsl on. It will just make the same assumptions that the browser does, by adding missing closing tags etc. I made a tool that extracted parts of web pages 10-15 years ago, and it worked well. There are of course cases where the html is so unstructured that the results were unpredictable, but it worked well in general.

pabs3 5y ago

I tend to reach for XPath selectors before CSS ones when querying HTML.

mro_name 5y ago

it's statically linkable rust, isn't it? Awesome. I'm looking for a successor to $ xmllint --html --xpath … that doesn't choke on inline svg.

teitoklien 5y ago

Maybe call it hq ?

Simplicitas 5y ago
My thoughts EXACTLY... but anyway, great new utility indeed!
- teitoklien 5y ago
  Haha, Indeed its a very good utility :D

srg0 5y ago

"htmlq: like jq, but for HTML" "jq is like sed for JSON data" sed: "While in some ways similar to an editor which permits scripted edits (_such as ed_), sed works by making only one pass over the input(s)" ed: "ed is a line-oriented text editor". Software definition through a reference to another software is somewhat confusing. Potential users come from different backgrounds (I had no idea what is jq), and it is not clear what are the defining features of each project. Is jq line oriented? Is htmlq operating in a single pass?

oauea 5y ago
What's this thing called a "computer" that people keep going on about, anyway?
- theandrewbailey 5y ago
  It's like a programmable loom, but for logical and mathematical operations.
  - samhw 5y ago
    You may be interested in the symbol grounding problem (https://en.wikipedia.org/wiki/Symbol_grounding_problem#Grounding_process https://en.wikipedia.org/wiki/Symbol_grounding_problem#Groun...). It's like the binding problem, but for symbols.
    - deleted 5y ago
      [deleted]
- da_chicken 5y ago
  It's a person who does mathematic calculations all day. For example, creating range tables for artillery, calculating averages or totals of a large range of values, or solving complex integrals or differential equations, and so on. They're commonly used in industry or government, especially in astronomy, aerospace and civil engineering for both simulation and analysis. Perhaps the most well-known computers were the Harvard Computers, which operated in the late 19th and early 20th centuries. As a job, computers were largely automated out of existence by solid-state transistor based automated computers and integrated circuit transistor automated computers in the 60s, 70s and 80s, which replaced the enormously expensive and often largely experimental electro-mechanical automated computers while radically reducing cost and improving performance both by several orders of magnitude.

abledon 5y ago

is anyone else using the https://github.com/json-path/JsonPath https://github.com/json-path/JsonPath over the jq route? I hope we standardize on some jq query language, like we have with a base set of SQL syntax

d--b 5y ago

Just being that guy: is there a reason you didn't call it hq?

zamadatix 5y ago
Not author but neither is the poster: Jq got away with it because it's one of the few 2 letter combinations that wasn't absolutely overloaded and "jquery" was already taken. OTOH nobody shortens HTML to H and HQ is an extremely common acronym, if not one of the most popular 2 letter acronyms you could pick.
- OJFord 5y ago
  jq didn't get away with it! Have you never tried searching for anything to do with it? How I wish it were called `jsonq`!
mgdm 5y ago
I just wanted to be slightly more descriptive and less likely to collide with other tools.
- notRobot 5y ago
  Hahah, I love how this is your second comment in 10 years on HN.
  - mgdm 5y ago
    Hah. Yeah. I had another account for a little while but then HN started to let me reset the password for this one quite recently, so here I am.

parhamn 5y ago

Ive been looking for a library that can find the best set of selectors to most consistently find the element youre looking for in a page. Any pointers to something that exists? Interestingly I've also found very little for dom extraction in the OS ML space.

Ronak123 5y ago

https://techflashes.com/top-upcoming-futuristic-technologies/ https://techflashes.com/top-upcoming-futuristic-technologies...

desktopninja 5y ago

Very nice tool. I've long spoiled myself with Powershell's: Invoke-WebRequest eg. # what is the latest release of apache-tomcat? $LINKS=$(Invoke-WebRequest -Uri 'https://tomcat.apache.org/download-80.cgi' | Select-Object -ExpandProperty Links) $LATEST=$($Links | Where-Object -Property href -Match '#8.5.[0-9]+').href.substring(1) $FETCH=$($Links | Where-Object -Property href -match "apache-tomcat-${LATEST}.zip$").href

Tepix 5y ago
Should it be $LINKS instead of $Links (2x)?
- desktopninja 5y ago
  "$links" works too because PWSH is not case sensitive. But I should have used $LINKS like you said for cleaner write-up.

necovek 5y ago

Nice, I expected something based on XPath (like xpd), but web developers dealing with HTML are infinitely more familiar with CSS selectors, so a great choice!

busterarm 5y ago
I want the option to use both, like Nokogiri gives you.
- necovek 5y ago
  Sure, that sounds nice, but having two simple tools each doing the job well in its own space is perfectly fine for me — do you imagine needing to combine Xpath and CSS queries in a single run?
  - busterarm 5y ago
    I've had to do it when dealing with some poorly-designed XML apis in the past. Nokogiri was a godsend.

who-shot-jr 5y ago

Good work!

oauea 5y ago

https://jsoup.org/ https://jsoup.org/ has been around for a long time and seems a bit more mature & maintained than this two-code-files 2-year-old repo. Highly recommend.

mcovalt 5y ago

I’d like to see a tool using lol-html [0] and their CSS selector API as a streaming HTML editor. [0] https://github.com/cloudflare/lol-html https://github.com/cloudflare/lol-html

chefandy 5y ago

If anyone is looking for a good library to do this in Python, PyQuery works well: https://pythonhosted.org/pyquery/ https://pythonhosted.org/pyquery/

elif 5y ago

When I saw the title I thought this was some machine learning-specific rmq/0mq message passing tech called HT. Very excited to zero.

soheil 5y ago

I'd use something like this script that you can put together yourself: #!/usr/bin/env ruby require 'nokogiri'; p Nokogiri::HTML(STDIN.read).css(ARGV[0]).text Just save it to a file in your /usr/local/bin/hq and do chmod +x !$ Then you can do: curl -s "https://news.ycombinator.com/news"|hq "tr:first-child .storylink" It uses Nokogiri[0], which is much more battle tested and works with CSS and XPath selectors. [0] https://nokogiri.org/tutorials/parsing_an_html_xml_document.html#from-a-string https://nokogiri.org/tutorials/parsing_an_html_xml_document....

drran 5y ago
Command just prints a bunch of text stitched together: curl -s "https://news.ycombinator.com/news"|hq "tr .storylink" Deploy a website on imgur.comMy £4 a month server can handle 4.2M requests a dayFirst Edition...
- soheil 5y ago
  Make sure you add :first-child as in my example, otherwise you'll get all the stories smooshed together.
  - drran 5y ago
    I want to see all stories, not first one only.

notorandit 5y ago

Next is xmlq: https://github.com/dscape/xmlq https://github.com/dscape/xmlq

downWidOutaFite 5y ago

Why? I find xpath's syntax much simpler and regular than jq's.

ludovicianul 5y ago

And a Java version with pre-compiled binaries: https://github.com/ludovicianul/hq https://github.com/ludovicianul/hq

bamdadd 5y ago

is there a brew install command ?

rendall 5y ago

What is jq?

jhatemyjob 5y ago

Crazy how a 300-line codebase manages to amass 2000 stars on Github and 700 upvotes on HN. Amazing ROI.

gravypod 5y ago

This looks really cool! I'd love to see a generic query language/tool library for structured data.

bandie91 5y ago

shameless self promotion: parsel[0] is a python script in front of the identically named python lib, and extracts parts of the HTML by CSS selector. the advantage of it compared to most similar tools is that you can navigate in the DOM tree up and down to find precisely what you want if the HTML is poorly marked up, or the searched parts are not close to each other. [0] https://github.com/bAndie91/tools/blob/master/usr/bin/parsel https://github.com/bAndie91/tools/blob/master/usr/bin/parsel