4 ms·

We have to crawl about 60-80k news websites per day [0]. I've spent about 1 month to test how scrapy could be a fit for our purposes. And, quite surprisingly,

We have to crawl about 60-80k news websites per day [0]. I've spent about 1 month to test how scrapy could be a fit for our purposes. And, quite surprisingly, it was hard to design a distributed web crawler. Scrapy is great for those in-the-middle tasks where you need to crawl a bit + process data on the go. We ended up just using requests to crawl the web. Then post-process the web pages in the next step. Many thanks to Zyte [1] (ex-ScrapingHub) for open-sourcing so many wonderful tools for us. I've spoke to Zyte's CEO, and was really fascinated how he still being a dev person while running such a big company. [0] https://newscatcherapi.com/news-api https://newscatcherapi.com/news-api [1] https://www.zyte.com/ https://www.zyte.com/

adamqureshi 5y ago
ok so i can just hire zyte to build me a custom scraper?
- artembugara 5y ago
  well, I think it is the cheapest & the fastest way, tbh.
  - adamqureshi 5y ago
    Thank you. The other site is also very interesting. I am working on this MVP and its news aggregator type site for a NICHE product. So i need to aggregate news for a brand from maybe 10-20 blogs and list the URL. thank you for sharing both. I'll reach out to them.
    - artembugara 5y ago
      I'm the co-founder of the other one, we could help you with your task. Feel free to contact me. artem [at] newscatcherapi.com
      - adamqureshi 5y ago
        Oh awesome! emailing you now. Thank you.
jcun4128 5y ago
> We have to crawl about 60-80k news websites per day [0] Can't even imagine that number... different languages or something?
- artembugara 5y ago
  Yeah, plus many of websites are quite niche news (construction news, for example)
1vuio0pswjnm7 5y ago
Does newscatcherapi provide a list of the 60-80K news sites so the customer knows what sources are actually being searched. Or even just way to determine if site X is among the 60-80K being searched.
- artembugara 5y ago
  Second one — yes. All sources, per request. But as a general rule, we crawl all we could/know which is a news website. Can add any niche/industry-specific websites on request.
m_ke 5y ago
Even with scrapy it's usually best to to crawl and store full html that can be scraped offline. It's too easy to miss something when scraping on the fly or drop data that might be useful in the future.
ramraj07 5y ago
How do you deal with these god awful SPAs though, when you just do requests..
- artembugara 5y ago
  We do it as a part of post-processing, we'd have to make one more call. Quite annoying though
- TechBro8615 5y ago
  SPAs tend to be some of the easiest sites to scrape (at least if you’re building targeted scrapers), because they come with a production-ready API out of the box. You can just watch the network tab of dev tools to know what endpoints to hit. Same applies to mobile apps – before you reach for BeautifulSoup, maybe check to see if the website has a mobile app too. It’s usually worth setting up a mitmproxy to see what API you might be able to scrape.
zepearl 5y ago
> I've spent about 1 month to test how scrapy could be a fit for our purposes. And, quite surprisingly, it was hard to design a distributed web crawler. Scrapy is great for those in-the-middle tasks where you need to crawl a bit + process data on the go. I agree (but in my case I needed 3 months to understand that :) ). I did start with plain "Scrapy" and it did work fine (maybe some parameters could be explained a bit better especially the ones involving concurrency) and it has been in general very helpful to initially understand some concepts related to web scraping. But now, after having rewritten multiple parts like 50 times to be able to handle most of what can happen, having split the workload into a multi-staged pipeline involving different specialized programs, having implemented custom code to distribute domains to be scraped to dedicated subprocesses (to greatly improve performance), and much more... I am using Scrapy basically to only download the raw bytes and to take care of what's mentioned in "robots.txt" => my next step will most probably be to replace it (now I don't like a lot having it as a dependency for my program, I therefore started thinking that such a complicated lib is a bit a "risk") with "requests" and something that can help to interpret "robots.txt". In my case ("wide" Internet scan involving so far ~8M webpages distributed over ~500K domains) I think that it might have been better to start straight with just using "requests".

adamqureshi 5y ago

ok so i can just hire zyte to build me a custom scraper?

artembugara 5y ago
well, I think it is the cheapest & the fastest way, tbh.
- adamqureshi 5y ago
  Thank you. The other site is also very interesting. I am working on this MVP and its news aggregator type site for a NICHE product. So i need to aggregate news for a brand from maybe 10-20 blogs and list the URL. thank you for sharing both. I'll reach out to them.
  - artembugara 5y ago
    I'm the co-founder of the other one, we could help you with your task. Feel free to contact me. artem [at] newscatcherapi.com
    - adamqureshi 5y ago
      Oh awesome! emailing you now. Thank you.

jcun4128 5y ago

> We have to crawl about 60-80k news websites per day [0] Can't even imagine that number... different languages or something?

artembugara 5y ago
Yeah, plus many of websites are quite niche news (construction news, for example)

1vuio0pswjnm7 5y ago

Does newscatcherapi provide a list of the 60-80K news sites so the customer knows what sources are actually being searched. Or even just way to determine if site X is among the 60-80K being searched.

artembugara 5y ago
Second one — yes. All sources, per request. But as a general rule, we crawl all we could/know which is a news website. Can add any niche/industry-specific websites on request.

m_ke 5y ago

Even with scrapy it's usually best to to crawl and store full html that can be scraped offline. It's too easy to miss something when scraping on the fly or drop data that might be useful in the future.

ramraj07 5y ago

How do you deal with these god awful SPAs though, when you just do requests..

artembugara 5y ago
We do it as a part of post-processing, we'd have to make one more call. Quite annoying though
TechBro8615 5y ago
SPAs tend to be some of the easiest sites to scrape (at least if you’re building targeted scrapers), because they come with a production-ready API out of the box. You can just watch the network tab of dev tools to know what endpoints to hit. Same applies to mobile apps – before you reach for BeautifulSoup, maybe check to see if the website has a mobile app too. It’s usually worth setting up a mitmproxy to see what API you might be able to scrape.

zepearl 5y ago

> I've spent about 1 month to test how scrapy could be a fit for our purposes. And, quite surprisingly, it was hard to design a distributed web crawler. Scrapy is great for those in-the-middle tasks where you need to crawl a bit + process data on the go. I agree (but in my case I needed 3 months to understand that :) ). I did start with plain "Scrapy" and it did work fine (maybe some parameters could be explained a bit better especially the ones involving concurrency) and it has been in general very helpful to initially understand some concepts related to web scraping. But now, after having rewritten multiple parts like 50 times to be able to handle most of what can happen, having split the workload into a multi-staged pipeline involving different specialized programs, having implemented custom code to distribute domains to be scraped to dedicated subprocesses (to greatly improve performance), and much more... I am using Scrapy basically to only download the raw bytes and to take care of what's mentioned in "robots.txt" => my next step will most probably be to replace it (now I don't like a lot having it as a dependency for my program, I therefore started thinking that such a complicated lib is a bit a "risk") with "requests" and something that can help to interpret "robots.txt". In my case ("wide" Internet scan involving so far ~8M webpages distributed over ~500K domains) I think that it might have been better to start straight with just using "requests".