3 ms·
Back when search engines caused this, the industry made an agreement and designed the robots.txt spec in order to avoid legal frameworks being made to stop them
by Cthulhu_ 2y ago
Back when search engines caused this, the industry made an agreement and designed the robots.txt spec in order to avoid legal frameworks being made to stop them. Because of that, legal frameworks weren't being made.
Now there's a new generation of hungry hungry hippo indexers that didn't agree to that and who feel intense pressure from competition to scoop up as much data as they can, who just ignore it.
Legislation should have been made anyway, and those that ignore robots.txt blocked / fined / throttled / etc.
- bitmasher9 2y agoI’m not sure that I like this plan. We shouldn’t let the illegal AIs gain more knowledge and usefulness than the legal ones. There’s other options besides a blanket ban.
- aqfamnzc 2y agoAs it is now, unethical AIs have a huge advantage over ethical ones.
- tremon 2y agoUnethical behaviour always has a huge advantage over ethical behaviour, that's nothing new and pretty much by definition. The only way to prevent a race to the bottom is to make the unethical behaviour illegal or unprofitable.
- joepie91_ 2y agoNone of the AIs have any 'knowledge' to begin with, so that's an easy one to satisfy.
- chneu 2y agoI'd be shocked if a single member of the US House or Congress know what robots.txt is.
- diggan 2y agoI'm pleased to declare you shocked: https://www.google.com/search?q=site%3Acongress.gov+%22robots.txt%22 https://www.google.com/search?q=site%3Acongress.gov+%22robot...
- skyyler 2y agoI don't know if "robots.txt" appearing in congressional record really counts. Do any of the decision makers appear to have a command of what the file does? Or do they typically relegate to industry professionals, as they often do?
- TheRealPomax 2y agoHow would legislation in the US or EU stop traffic from China or Thailand or Russia? At best you'd be fragmenting the internet, which isn't really a "best", that's a terrible idea.
- j2kun 2y agoThis is the key point, but if US laws are being violated and AI is considered part of national security, that could be used by the US government in international negotiations, and for justification for sanctions, etc. It would be a good deterrent.