4 ms·
Thanks. Hoes does WBM get the page? Does the economist let it's crawlers through?
by retube 5y ago
Thanks. Hoes does WBM get the page? Does the economist let it's crawlers through?
- jeroenhd 5y agoCrawlers are often permitted to get more web visibility. I think the Web Archive also crawls the web as a browser sometimes, so they can't always distinguish between crawlers. That said, with NoScript on I can just read the economist article, no banners or anything.
- Cthulhu_ 5y agoI do wonder; at some point, Google said they would penalize websites that show their crawlers something else than real visitors.
- flerchin 5y agoJust disable javascript and the entire article loads on the economist.