5 ms·
Pup: Parsing HTML at the command line
- turtlebits 4y agoWill check it out, but would have preferred XPath selectors instead of CSS.
- wmichelin 4y agoWhy? CSS selectors are the normal web developer way to select content from a document. Even JavaScript adopted the approach.
- tuukkah 4y agoXPath supports more complex queries. In JavaScript, XPath is available as document.evaluate
- undume 4y agoxmllint can do that: curl example.org | xmllint --html --xpath '//some/xpath/selector' -
- jwilk 4y agoOnly if it's good old HTML 4. libxml2's parser doesn't grok HTML 5.
- BossHogg 4y agoThere's also https://github.com/charmparticle/xpe https://github.com/charmparticle/xpe
- natrys 4y agoAlso: https://github.com/benibela/xidel https://github.com/benibela/xidel
- curben 4y agoI'm using xmlstarlet in Alpine as a bare minimum way to scrap a webpage in CI pipeline.
- ludovicianul 4y agoThis has support for both XPath and CSS selectors: https://github.com/ludovicianul/hq https://github.com/ludovicianul/hq
- bijoo 4y agoIt looks like the project became inactive for a bit and there are alternatives such as htmlq, etc. https://github.com/ericchiang/pup/issues/150 https://github.com/ericchiang/pup/issues/150
- mananaysiempre 4y agoFrom the looks of it, htmlq doesn’t have anything comparable to pup’s JSON output. That JSON is cumbersome to work with, but combined with jq it allows one to extend the shell hackery just a little bit beyond what CSS can do.
- wwader 4y agoHey, i'm the author of fq. It can convert to/from html and JSON (in two different modes). Use -d html, or the fromhtml, fromxml and toxml functions. Ex: $ curl -s https://news.ycombinator.com/ | fq -r -d html 'grep_by(."@class"=="titleline").a."#text"' Inkbase: Programmable Ink New details on commercial spyware vendor Variston How We Built Fly Postgres ... $ curl -s https://news.ycombinator.com/ | fq -r -d html '{hosts: {host: [grep_by(."@class"=="titleline").a."@href" | fromurl.host]}} | toxml({indent:2})' <hosts> <host>www.inkandswitch.com</host> <host>blog.google</host> <host>fly.io</host> ... </hosts> See https://github.com/wader/fq/blob/master/doc/formats.md#xml https://github.com/wader/fq/blob/master/doc/formats.md#xml and https://github.com/wader/fq/blob/master/doc/formats.md#html https://github.com/wader/fq/blob/master/doc/formats.md#html for examples and documentations.
- John23832 4y agoWhile this is nice, it's three years old. Direct installation of brew scripts isn't supported anymore. `go get` installs aren't either. It needs an update.
- JodieBenitez 4y agoIt may need an update, but not for installation: go install github.com/ericchiang/pup@latest
- thangalin 4y agohttps://www.w3.org/Tools/HTML-XML-utils/ https://www.w3.org/Tools/HTML-XML-utils/
- bsnnkv 4y agoI use this extensively in bash scripts where I need to scrape HTML reliably and consistently. Cannot recommend it highly enough. In fact, all of the source data for a project of mine, Baytyab[1] (couplet-finder) was scraped using bash + pup. [1]: https://baytyab.com https://baytyab.com
- pipeline_peak 4y agoWould you recommend it over BeautifulSoup?
- bsnnkv 4y agoI'd pick any CLI I can use directly in bash vs having to write Python to use a lib any day of the week.
- culi 4y agoI had a recent realizing that Deno is actually an amazing tool for scripts and webscraping. It runs TS natively so I can take advantage of TypeScript's amazing type system when interacting with complex APIs. Being able to import libraries from a url also makes it a breeze to use any libraries (I've yet to run into a node module that Deno's std/node library wasn't able to polyfill). I can basically write a .ts file anywhere and run it from the CLI with deno. Definitely prefer it over python at this point
- brigandish 4y agoI use Deno for linting, having a binary there instead of using NPM to get a linter is a different level of simple. I like your idea, though I rarely use js for scripts I'm sure it'll come in handy.
- bryanrasmussen 4y agoIs there any situation where you end up using the type system for interacting with APIs but also have to webscrape on the same service? I am interested in hearing more about this as maybe I should switch from Node to Deno for a scraping project I have.
- ducktective 4y agoI'm looking for something like this but for scraping SPAs and JS-rich web content. A single static binary like this not chrome driver or selenium etc
- mplewis9z 4y agoGood luck with that - you'd need to have a fully standards-compliant HTML/CSS/JS engine in order to get the same output, and at that point that's literally just a browser (and would require a massive investment of time for maintenance). A driver for an existing browser is the only reasonable option for the foreseeable future, with how complex modern web development has become.
- chrismarlow9 4y agoYou need something to replace the curl part of this, not pup. You can pipe js into chrome via CLI with --repl flag and dump the full Dom and then pipe that into pup. You'll be stuck with GET requests only though, and very simple ones, unless you get creative with the js you pipe in.
- dang 4y agoRelated: Pup – Like Jq, but for HTML - https://news.ycombinator.com/item?id=24797697 https://news.ycombinator.com/item?id=24797697 - Oct 2020 (2 comments) Show HN: Pup – A command-line HTML parser - https://news.ycombinator.com/item?id=8312249 https://news.ycombinator.com/item?id=8312249 - Sept 2014 (27 comments) Random bit of history: that Show HN was a very early choice for what is now called the second-chance pool: Ask HN: Why did three HN stories jump 100 ranking points in 5 mins? - https://news.ycombinator.com/item?id=8313505 https://news.ycombinator.com/item?id=8313505 - Sept 2014 (6 comments)
- abathur 4y agoEnjoying pup recently myself. Used it for a little demo/post on shell packaging approaches in Nix this summer https://t-ravis.com/post/shell/no_look_no_leap_shell_with_nix/ https://t-ravis.com/post/shell/no_look_no_leap_shell_with_ni...
- panic 4y agoIs there a tool like this that can modify part of the HTML without changing the formatting or inserting new tags unnecessarily? I know it's a difficult problem given the complexity of HTML parsing, but I'd like to be able to work with parts of pages programmatically without messing up the hand-written formatting in other parts.
- k__ 4y agoI'd go so far and assume it's an AI problem.
- mh- 4y agoProbably not without constructing a DOM. Which nowadays, in practice, means using Blink or WebKit.
- yencabulator 4y agohttps://github.com/cloudflare/lol-html https://github.com/cloudflare/lol-html is a HTML rewriter that explicitly does not construct DOM or an AST, but streams nodes throughs ASAP. https://blog.cloudflare.com/html-parsing-1/ https://blog.cloudflare.com/html-parsing-1/ https://blog.cloudflare.com/html-parsing-2/ https://blog.cloudflare.com/html-parsing-2/
- mh- 4y agoThis is the best approach if they don't need to consider client-side JS mutations. Not sure why I interpreted it that way in my original reply. Thanks!
- jakeogh 4y agoAnything out there that lets one programatically query the style on an element? I get a DOM object from lxmlhtml.fromstring(), but haven't figured out how to get element styles, so I can display something in bold, or apply a term color.
- culi 4y agoif those styles are defined inline it should be straightforward but most likely there's external styles being applied which might be a nontrivial problem to solve However we do have document.defaultView.getComputedStyle which probably will solve most of your needs. Not sure if that API is available with this tool though
- ghoul2 4y agoIs there a tool to transform html? Simple example, select all img tags without alt text, and insert given text into the alt tag. Or change the domain for all a.hrefs
- fcholf 4y agoYou may use xslt for this kind of operation (https://en.wikipedia.org/wiki/XSLT https://en.wikipedia.org/wiki/XSLT) but it is often overkilled. I would love a language that allows to easily write transductions of HTML documents but I think that combining simplicity and expressiveness for this kind of tasks is a bit painful. I usually end up using pandoc filters or some dedicated javascript script that transforms the DOM and output a new HTML file.
- sorenjan 4y agoLike others have said, pup seems to be abandoned. It's surprising that there isn't a well supported standard tool to work with html in the terminal, like jq for json. Last time I looked at using pup or similar I wanted to extract two values for each element. For example, let's say I have the following html: <div class="image"> <p>Sunset in Hawaii</p> <img src="../randomstring123.jpg"> </div> <div class="image"> ... Now I'd like to get both the image description, and the source, for each similar image in the page. Preferably so I can pipe it to curl and do curl -o "$description.jpg" "$url" I couldn't find an easy way of doing it, so I used Python instead.
- msla 4y agoYou could probably sprinkle a few more uses of cat into your command lines, but otherwise it's fine.