4 ms·

How do you get news from its sources?

• caballeto 6y ago
  First, I extract the links to the news articles by searching them on the home page or sitemaps. Then, for each article I scrap raw html and extract key things like title, description, publish date, authors, etc. Usually each news article has a schema embedded within it, so it greatly simplifies things of extracting article details.
  • erwinkle 6y ago
    Do you scrape using web requests? Or do you use headless browsing to load the page and extract the contents out of the DOM?
    • caballeto 6y ago
      I use standard web requests to get raw html and parse the data from it, no headless browsing. I guess all the news pages I've met so far were static.