5 ms·
Aspider: A lightweight, asynchronous micro-framework based on asyncio
- akdor1154 8y agoA "lightweight" "micro-framework" that downloads and runs Chrome in order to make an http request? :/
- subway 8y agoHey, at least they aren't bogged down by silly things like controlling what version of deps get installed. aspider depends on pyppeteer (with no version pinning) to download a copy of chromium (with no checksum validation).
- vosper 8y agoIt's for scraping, as far as I know the best way to do that these days is with headless Chrome, and then probably injecting some custom JS. What this provides over just using puppeteer or pyppeteer directly I'm not sure...
- drenvuk 8y agoIs there a better way to scrape nowadays? There's so much javascript that needs to be loaded and run to make a site usable. Scrapy doesn't really do enough anymore.
- carlsborg 8y agoI think Puppeteer also downloads a recent version of Chrome/Chromium that is guaranteed to work with the API. Why is this out of the ordinary for a headless web scraping framework?
- lilbobbytables 8y agoFramework feels like an odd choice of words. I would call it an async web spider library or async web scraping library. When I hear framework I think client interaction - whether cli, API, or web page.
- howie6879 8y agoHi lilbobbytables: Thanks for your suggestion, I have modified it.
- howie6879 8y agoHi akdor1154: This download will only be triggered if you crawl the page loaded by js.
- carlsborg 8y agoTitle should mention that its a web scraping framework
- howie6879 8y agoI am so sorry, you are right, it is my problem.