2 ms·
Robots.txt was created long before Google and before people were thinking about SEO: https://en.wikipedia.org/wiki/Robots.txt https://en.wikipedia.org/wiki/Rob
by PaulHoule 1y ago
Robots.txt was created long before Google and before people were thinking about SEO:
https://en.wikipedia.org/wiki/Robots.txt https://en.wikipedia.org/wiki/Robots.txt
The scenario I remember was that the underfunded math department had an underpowered server connected via a wide and short pipe to the overfunded CS department and webcrawler experiments would crash the math department's web site repeatedly.
- LorenPechtel 1y agoYup. Robots.txt was a don't-swamp-me thing.
- abirch 1y agoWith the advent of AI and the notion of actually going to a website as being quaint: each website should have a humans.txt such as https://www.netflix.com/humans.txt https://www.netflix.com/humans.txt or https://www.google.com/humans.txt https://www.google.com/humans.txt
- PaulHoule 1y agoWhat everybody is missing is that AI inference (not training) is a route out of the enshittification economy. One reason why Cloudflare is harassing you all the time to click on traffic lights and motorcycles is to slam the door from some of the exit routes.
- EPendragon 1y agoI have not heard of humans.txt before. It is apparently used for acknowledgement and crediting the dev team who created the resource.
- EPendragon 1y agoIt is so interesting to track this technology's origin back to the source. It makes sense that it would come from a background of limited resources where things would break if you overwhelm it. It didn't take much to do so.