3 ms·
Show HN: flyscrape – An expressive and elegant web scraper
- dinkleberg 3y agoThis looks interesting, how would you compare it with colly? I’ve used colly for a while and overall am pleased with it, but things can always be improved.
- philippta 3y agoColly is a great library for builing scrapers if you are familiar with Go. Flyscrape a cli tool that lets you define your scraping logic with a declarative JavaScript API, while it takes care of issuing requests, caching, rate limiting and more.
- dinkleberg 3y agoAh lol somehow I totally missed the fact that you write it with JS. That is a neat approach.
- peyloride 3y agoHey, quick question: Is there a way to handle async behavior? Like in puppeteer we had something like waitFor helper to wait while loading the page. I couldn't find a similar thing in your readme. Or is this framework for only static pages?
- philippta 3y agoAt this time it works with server-side rendered / static pages only and does not execute any JavaScript of the website. The data extraction function of your script also runs aside the crawler, not inside the page.