5 ms·

Web Environment Integrity? https://en.m.wikipedia.org/wiki/Web_Environment_Integrity https://en.m.wikipedia.org/wiki/Web_Environment_Integrity

dcow 3y ago
Say a site uses WEI to block Google’s crawlers. If Google actually respected that then maybe it’s a fair playing field. But knowing Google they’ll issue their crawlers valid certs and create a huge dystopian nightmare.
- extraduder_ire 3y ago
  IIRC, it's in the spec to have WEI fail some of the time randomly, so that sites are designed to deal with real failures. Though, that may just be in there to make the spec less unappealing.
- google234123 3y ago
  What? Google has respected robot.txt for ever, why do you think it will change now?
  - ipaddr 3y ago
    It doesn't always and often returns with no agent from a google ip.
    - notatoad 3y ago
      is there any evidence of this that you can share?
- mymacmachine 3y ago
  I'd imagine a different dystopian nightmare. Google don't care if an individual site blocks them (in my experience they are one of the better crawlers at obeying robots.txt) because they know - given their search monopoly - that the vast majority of the internet would never dare to cut them out. WEI is a way to cement their current position further without needing to play dodgy games with special Google certs. If they put out WEI and start encouraging sites to use it for better SEO, the majority of the internet will do it (and wont put in the same effort to support other crawlers because those search pages are only < 1% of traffic according to their Google Analytics) At that point, when change (legal, technical, social, whatever) comes to try and break apart their hold on search - there is yet another technical constraint that works in Google's favour. It won't be Google's "fault" that websites give them privileged crawl access and greater reach, but we'll all be stuck with it.
  - Vriska1 3y ago
    [dead]
- ChadNauseam 3y ago
  I think almost no site would do that. So I don't see why google would risk the political consequences of doing that

dcow 3y ago

Say a site uses WEI to block Google’s crawlers. If Google actually respected that then maybe it’s a fair playing field. But knowing Google they’ll issue their crawlers valid certs and create a huge dystopian nightmare.

extraduder_ire 3y ago
IIRC, it's in the spec to have WEI fail some of the time randomly, so that sites are designed to deal with real failures. Though, that may just be in there to make the spec less unappealing.
google234123 3y ago
What? Google has respected robot.txt for ever, why do you think it will change now?
- ipaddr 3y ago
  It doesn't always and often returns with no agent from a google ip.
  - notatoad 3y ago
    is there any evidence of this that you can share?
mymacmachine 3y ago
I'd imagine a different dystopian nightmare. Google don't care if an individual site blocks them (in my experience they are one of the better crawlers at obeying robots.txt) because they know - given their search monopoly - that the vast majority of the internet would never dare to cut them out. WEI is a way to cement their current position further without needing to play dodgy games with special Google certs. If they put out WEI and start encouraging sites to use it for better SEO, the majority of the internet will do it (and wont put in the same effort to support other crawlers because those search pages are only < 1% of traffic according to their Google Analytics) At that point, when change (legal, technical, social, whatever) comes to try and break apart their hold on search - there is yet another technical constraint that works in Google's favour. It won't be Google's "fault" that websites give them privileged crawl access and greater reach, but we'll all be stuck with it.
- Vriska1 3y ago
  [dead]
ChadNauseam 3y ago
I think almost no site would do that. So I don't see why google would risk the political consequences of doing that