4 ms·
I think you'd be surprised by the amount of website you can scrape without an headless browser. Even Google SERP can be scraped with a simple HTTP client.
by daolf 6y ago
I think you'd be surprised by the amount of website you can scrape without an headless browser.
Even Google SERP can be scraped with a simple HTTP client.
- tyingq 6y agoThat's what I meant by narrow...a known set of sites and data you want to extract. I imagine, for example, building on the SERP example might hit a wall if you added logged in vs not logged SERPS, iterating over carousel data, reading advertisement data etc.
- daolf 6y agolog in wall can easily be bypassed with an HTTP client by setting the correct auth header. From what I can observe, 2/3 websites can be scraped without using a headless browser.
- hakube 6y agoHow do you deal with JavaScript then?