4 ms·
Last year I needed some quick scraping and I used a headless Chromium to render webpages and print the HTML then analyze it with C#. I don't remember exactly,
by jmnicolas 5y ago
Last year I needed some quick scraping and I used a headless Chromium to render webpages and print the HTML then analyze it with C#.
I don't remember exactly, but I think it was around 100 or 200 loc, so not exactly something that took long to write.
In fact the most difficult thing was to figure how to pass the right args to Chromium.
I wonder what does a scraping framework offer?
- Veen 5y ago> I wonder what does a scraping framework offer? HTTP requests, HTML parsing, crawling, data extraction, wrapping complex browser APIs etc. Nothing you couldn't do yourself, but like most frameworks, they abstract the messy details so you can get a scraper working quickly without having to cobble together a bunch of libraries or re-invent the wheel.
- jmnicolas 5y agoI see thanks.
- jrochkind1 5y agoJust for one example, when you have to get a form, and then submit the form, with the CSRF protection that was in the form... of course you COULD write that yourself by printing HTML and then analyzing it with C# (which triggers more requests to chromium I guess), but you're probably going to wonder why you are reinventing the wheel when you want to be getting on to the domain-specific stuff.
- jmnicolas 5y agoAh yes I see. Mine was read-only, so no need for complex stuff.
- elorant 5y agoThrottling is a prime example. If you start loading multitudes of sites in asynchronous fashion you'll have to enter some delay otherwise you run the risk of choking the server in misconfigured sites. I've DDoSed sites accidentally this way. You can of course build a framework on your own, and that's pretty much what every scraper does eventually, but it takes time and a lot of effort.