4 ms·
One part of the "web's contract" is that googlebot and other scrapers should get what the users get. And vice versa - if you're telling googlebot that there's a
by PeterisP 7y ago
One part of the "web's contract" is that googlebot and other scrapers should get what the users get. And vice versa - if you're telling googlebot that there's a particular text available to the public; I'd expect to get the same content that you just gave googlebot and not something else, thank you very much.
- jacquesm 7y agoThat used to be the case. But Google has long ago given certain sites a pass on that.
- Forge36 7y agoDid they? This fight against Incognito mode suggests otherwise
- kpozin 7y agoEver try to click on a LinkedIn profile from a search result? Most of them just show a LinkedIn login screen, and none of the text from the search result's snippet.
- Mindwipe 7y agoShocking I know, but it seems like Google contradicting other parts of Google due to a lack of strategic leadership, which almost never happens. Daily...
- quantumwannabe 7y agoGoogle will index paywalled content from news sites provided they annotate it correctly. [1] [1] https://developers.google.com/search/docs/data-types/paywalled-content https://developers.google.com/search/docs/data-types/paywall...
- crazygringo 7y agoWhat do scrapers and indexing engines have to do with it? Is it normal for them to run a full headless browser in incognito mode? I was under the impression that they're either just doing straight HTTP requests for HTML only... or they're running a full headless browser in normal mode. So I'm not getting what's different here? Sites have a long history of serving up different content to different users, e.g. to paying users, or blocking certain countries based on content contracts. It's certainly not part of the "web's contract" that scrapers get paywalled content they haven't paid for.