5 ms·
Does it seem to anyone like eventually the entire internet will be login only? At this point knowledge seems to be gathered and replicated to great effect and
by hardwaresofton 1y ago
Does it seem to anyone like eventually the entire internet will be login only?
At this point knowledge seems to be gathered and replicated to great effect and sites that either want to monetize their content OR prevent bot traffic wasting resources seem to have one easy option.
- mjevans 1y agoStatic, Near Static (not generated on demand at least; generated only on real content update), and Login seems likely. AI not caching things is a real issue. Sites being difficult TO cache / failing the 'wget mirror test' is the other side of the issue.
- grishka 1y agoWhat about AI not respecting robots.txt? I myself have never ran into this, but I've seen complaints of many people who did.
- tonyhart7 1y ago"What about AI not respecting robots.txt?" since when actor that want gather your entire data respect things like this??? how can you enforce such things with just "please don't crawl this directory thanks"
- arkh 1y ago> how can you enforce such things A van, some balaclavas and 4 people with big sticks.
- grishka 1y agoYou can't enforce them, but for the entire preceding history of the internet, most crawlers respected them. But then AI happened and those companies decided that their noble mission of forcing their slop into as many facets of human life as possible is above some stupid rules.