5 ms·
Sure. But you can't use jq to scrape arbitrary websites, for example. :)
by sillysaurus3 9y ago
Sure. But you can't use jq to scrape arbitrary websites, for example. :)
- jldugger 9y agoOf course, if it's anything like HTML, the formatting will vary over time that you really want a more permissive parser like BeautifulSoup. I haven't found a cli interface, so I briefly wrote my own ages ago: https://github.com/jldugger/dotfiles/blob/master/bin/select.py https://github.com/jldugger/dotfiles/blob/master/bin/select.....
- nurettin 9y agoFor cases where a website is not a tutorial for websites, regex is a suitable tool for scraping.
- dpflan 9y agoJeff Atwood has an entertaining post about parsing HTML with regular expressions: https://blog.codinghorror.com/parsing-html-the-cthulhu-way/ https://blog.codinghorror.com/parsing-html-the-cthulhu-way/ “”” That's right, if you attempt to parse HTML with regular expressions, you're succumbing to the temptations of the dark god Cthulhu's … er … code. “””
- alexozer 9y agoLet's not forget about this masterpiece: https://stackoverflow.com/a/1732454/864310 https://stackoverflow.com/a/1732454/864310
- dpflan 9y agoIndeed, its quality cannot be ignored and must be shared; it’s referenced in the Atwood post.
- ams6110 9y agoParsing and scraping are different things though. You don't need to parse a web page to extract specific things from it.