3 ms·
Instead of beautifulsoup you're honestly better off just running an instance of Firefox/Chrome and scraping data that way. You can do fairly easily these days a
by dom96 5y ago
Instead of beautifulsoup you're honestly better off just running an instance of Firefox/Chrome and scraping data that way. You can do fairly easily these days and I have a library that allows you to do so in Nim: https://github.com/dom96/webdriver https://github.com/dom96/webdriver
- MR4D 5y agoI use it to parse crappy (non-standard) RSS feeds offline for an internal app I use. chrome-scraping would just be more of a pain (although I've done it before).
- elcritch 5y agoHave you tried htmlparser? Its fairly tolerant of "wild" html. It could use some tweaks to match the html5 spec that defines a lot more tags that don't require end tags. Also the library isnt to bad to grok and tweak yourself. There's also Nimquery that looks nice, though I haven't used it yet. Still looks to replace a lot of beautifulsoup.
- MR4D 5y agoNo, but I use feedparser in the same script. Basically I'm scraping SEC data, but the script has been around for awhile untouched, so there are probably better solutions today.