3 ms·
I think a version of this is what the internet needs but using headless browsers from the client and with a somewhat-centrally curated set of scraper "recipes"
by kodablah 5y ago
I think a version of this is what the internet needs but using headless browsers from the client and with a somewhat-centrally curated set of scraper "recipes" if you will. Basically a community curated/updated set of scraper logic per site (yes some trust is required) that essentially provides JSON data and/or APIs based on the site. Even just a neutered HTML equivalent of sites (e.g. amp w/out the Google and ads stuff) would be good.
Since it is all client side, it can be dubbed a "browser" not a "scraper" and one might hope popularity is high enough that active blocking of it is blatantly user hostile. Granted one hopes that, like EasyList and uBO and others have shown, the community can outpace site owners. Not appearing headless (tunneling captchas, literal mousemove events in pseudo-random human-like ways, etc) should be doable.
It's something I have thought about and once dubbed "recapitate" (https://github.com/cretz/software-ideas/issues/82 https://github.com/cretz/software-ideas/issues/82) and plan to revisit. I have seen many versions of this attempted. We need to encourage shared data extraction tools.