2 ms·
> If I was US gov. I would've made law in the early 1990s that says every website, blog etc. is private property and no entity has right to crawl it if not give
by blibble 5y ago
> If I was US gov. I would've made law in the early 1990s that says every website, blog etc. is private property and no entity has right to crawl it if not given permission basically eliminating early Internet Search Engine industry meaning the behemoth like Google would probably never exist.
it would have existed, just outside of the US
- mrkramer 5y agoCrawling a website in my opinion is akin to someone snooping inside your house and making a index of rooms and stuff you have. Robots.txt protocol exists but it is not even official internet standard under IETF. You can break robots.txt rules and there are no consequences at all. If robots.txt protocol was made law then it would be whole another story.