2 ms·
First, I extract the links to the news articles by searching them on the home page or sitemaps. Then, for each article I scrap raw html and extract key things l
by caballeto 6y ago
First, I extract the links to the news articles by searching them on the home page or sitemaps. Then, for each article I scrap raw html and extract key things like title, description, publish date, authors, etc. Usually each news article has a schema embedded within it, so it greatly simplifies things of extracting article details.
- erwinkle 6y agoDo you scrape using web requests? Or do you use headless browsing to load the page and extract the contents out of the DOM?
- caballeto 6y agoI use standard web requests to get raw html and parse the data from it, no headless browsing. I guess all the news pages I've met so far were static.