5 ms·
And yet I constantly get CAPTCHA's just from blocking tracking cookies. The biggest scraper of them all goes to incredible lengths to prevent scraping. How iro
by throwaway189262 6y ago
And yet I constantly get CAPTCHA's just from blocking tracking cookies.
The biggest scraper of them all goes to incredible lengths to prevent scraping. How ironic.
I hope the US govt demolishes these monopolies. It's not just in web either, the closest historical precedent I can think of is the Robber Barons of the 30's
- ehnto 6y agoIt's them protecting their technology moat I figure, not so ironic, more of a hoarder protecting their hoard.
- Google234 6y agoThere’s a thing called a robot.txt.
- xondono 6y agoYes, and google has been observed several times to ignore the robots.txt file
- Google234 6y agoNo, this is true in general. Maybe there’s been some bugs (I would be surprised, can you link these multiple examples?) but Google respects robot.txt files
- pkilgore 6y agoEven assuming Google would not ignore that for a popular enough site (they would know!), there is no robot.txt setting that says "index my site, but please don't steal my content, thanks."
- throwaway189262 6y agoYeah then they won't index you at all and you lose all your traffic instantly. How about a setting to index my site without scaping my content and using it all over the place without attribution