3 ms·
If I understand correctly this requires a shift from avoiding certain domains to avoiding certain URLs. At present myhometownnewspaper.com, i.e., newspapers, g
by 1vuio0pswjnm7 3y ago
If I understand correctly this requires a shift from avoiding certain domains to avoiding certain URLs.
At present myhometownnewspaper.com, i.e., newspapers, generally have a robots.txt which lists a "sitemap.xml". The sitemap should list all the articles but will not list /allurdatabelongtous.
Ads and analytics URLs do not appear in "sitemap.xml". Maybe some of the ads domains are in ads.txt if it exists. In any event, it's easy to avoid the garabage URLs. IMHO, a web browser that auto-loads resources is not the best software for retrieving the URLs in a sitemap.