3 ms·
You may find the amp pages slightly easier to scrape. i.e. This: https://www.bbc.co.uk/news/amp/health-54795657 https://www.bbc.co.uk/news/amp/health-54795657
by Jaruzel 6y ago
You may find the amp pages slightly easier to scrape.
i.e. This: https://www.bbc.co.uk/news/amp/health-54795657 https://www.bbc.co.uk/news/amp/health-54795657
vs This: https://www.bbc.co.uk/news/health-54795657 https://www.bbc.co.uk/news/health-54795657
I'm working on a similar thing at moment (BBC html -> markdown) so also exploring the best way to do it.
- comice 6y agonice thanks, that looked promising but I've checked a number of older articles and they have no amp version (despite the HTML for the site referencing an amp version, the url is 404!) I'll see if I can figure out if all newer pages will have permanent amp versions, or whether all amp versions drop away over time.