4 ms·
Ah, htmlq [1] is a missing one that's not on the list! Straight from the repo: "Like jq, but for HTML." I find it useful for quickly hacking scripts together
by zamubafoo 5y ago
Ah, htmlq [1] is a missing one that's not on the list!
Straight from the repo: "Like jq, but for HTML."
I find it useful for quickly hacking scripts together and exploring data. Very useful for the iterative process of finding good CSS selectors with the data that I can get without javascript running.
---
[1] https://github.com/mgdm/htmlq https://github.com/mgdm/htmlq
- ducktective 5y agoI use pup for this. People who tried both, any difference? https://github.com/EricChiang/pup https://github.com/EricChiang/pup
- zamubafoo 5y agoNot sure if pup supports this but something I do use fairly often (and copied into my own internal tooling) is the ability to filter out results as a flag in the CLI. For example, something I usually do is: curl --include --location https://example.com | tee /tmp/example-com.html | htmlq --base https://example.com a --attribute href --remove-nodes 'a[href*="#"],a[href^="javascript"],a[href*="?"]' This grabs the page, shunts a copy to /tmp for subsequent, iterative testing, then tries to grab all the links while filtering out any links that have a '#', '?', or start with the word 'javascript'. This is super helpful when I'm just exploring some HTML scrape and trying to build a graph of links without having to pop out a proper programming language just yet.