3 ms·
Resource efficiency has a lot to do with that, I think. Crawling sites twice will add not only to the burden of the bots but the sites too. Admins may disapprov
by ejr 12y ago
Resource efficiency has a lot to do with that, I think. Crawling sites twice will add not only to the burden of the bots but the sites too. Admins may disapprove of this.
I wonder if users can be motivated to do the checking themselves when they post the link.
- quesera 12y agoCould try https first, then. While it's possible to serve completely different pages and sites on https vs http, it's rare in practice, and would be reasonable to penalize.
- embolalia 12y agoI think that's absolutely reasonable to penalize. From a technical perspective, returning a different resource depending on the scheme of the URI is absolutely acceptable, but from a UX standpoint it's ridiculous to have a difference between http and https.