3 ms·
Can you talk about how you avoid getting blocked by captcha? Do you use RSS feeds? In that case do you not return images?
by moneywoes 6y ago
Can you talk about how you avoid getting blocked by captcha? Do you use RSS feeds? In that case do you not return images?
- caballeto 6y agoThe scraper only scraps the recent content, so it avoids scraping same links. As the daily number of articles published by the website is not very big, I make requests with a break in between, total number of requests is not that big, so the ip is not blocked. Also, adding proxy servers is a possible solution, i.e. having a proxy pool to proxy requests. I guess in this scenario it is easy to by pass capcha.
- TechBro8615 6y agoNitpick: it’s “scrapes,” not “scraps.” To scrap something is to throw it away. To scrape something is to ingest it. The two words are, in fact, nearly opposite. Sorry — I don’t mean to pick on you — it’s just that I see this mistake with an inexplicably high frequency, and it’s like nails on a chalkboard for me. Also, very cool service!
- caballeto 6y agoThanks, will take take a note for future.
- caballeto 6y agoNo, I don't use RSS feeds. I extract news links from the home page and sitemaps mainly. Most of the news articles have a news schema embedded within them. You can check info about it at schema.org. This is a piece of JSON which contains the description of the article, usually title, publish date, description & image links.