3 ms·
The scraper only scraps the recent content, so it avoids scraping same links. As the daily number of articles published by the website is not very big, I make r
by caballeto 6y ago
The scraper only scraps the recent content, so it avoids scraping same links. As the daily number of articles published by the website is not very big, I make requests with a break in between, total number of requests is not that big, so the ip is not blocked. Also, adding proxy servers is a possible solution, i.e. having a proxy pool to proxy requests. I guess in this scenario it is easy to by pass capcha.
- TechBro8615 6y agoNitpick: it’s “scrapes,” not “scraps.” To scrap something is to throw it away. To scrape something is to ingest it. The two words are, in fact, nearly opposite. Sorry — I don’t mean to pick on you — it’s just that I see this mistake with an inexplicably high frequency, and it’s like nails on a chalkboard for me. Also, very cool service!
- caballeto 6y agoThanks, will take take a note for future.