3 ms·
Some people just believe that because someone says so, everyone will nicely obey and follow the rules, don't know maybe it is a cultural thing.
by pjmlp 1y ago
Some people just believe that because someone says so, everyone will nicely obey and follow the rules, don't know maybe it is a cultural thing.
- vintagedave 1y agoOr a positive belief in human nature. I admit I'm one of those people. After decades where I should perhaps be a bit more cynical, from time to time I am still shocked or saddened when I see people do things that benefit themselves over others. But I kinda like having this attitude and expectation. Makes me feel healthier.
- tuyiown 1y agoI deeply agree with you, and I'd like to add: Trust by default, also by default, never ignoring suspicious signals. Trust is not being naïve, I find the confusion of both very worrying.
- Sammi 1y agoYou don't have to go as far as to straight up "trust by default". You can instead "give a chance" by default, which is the middle path. Actually Veritasium has a great video about this. It's proven as the most effective strategy in monte carlo simulation. EDIT: This one: https://youtu.be/mScpHTIi-kM https://youtu.be/mScpHTIi-kM
- chasd00 1y agoi like that Veritasium vid a lot, i've watched it a couple times. The thing is, there's no way to retaliate against a crawler ignoring robots.txt. IP bans don't work, user agent bans don't work, there's no human to shame on social media ether. If there's no way to retaliate or provide some kind of meaningful negative feedback then the whole thing breaks down. Back to the Veritasium video, if a crawler defects they reap the reward but there's no way for the content provider to defect so the crawler defects 100% of the time and gets 100% of the defection points. I can't remember when i first read the rfp for robots.txt but I do remember finding it strange that it was a "pretty please" request against a crawler that has a financial incentive to crawl as much as it can. Why even go through the effort to type it out? EDIT: i thought about it for a min, i think in the olden days a crawler crawling every path through a website could yield an inferior search index. So robots.txt gave search engines a hint on what content was valuable to index. The content provider gained because their SEO was better (and cpu util. lower) and the search engine gained because their index was better. So there was an advantage to cooperation then but with crawlers feeding LLMs that isn't the case.
- Sammi 1y agoNo robots.txt can't fix this. Have you tried Anubis? It was all over the internet a few months ago. I wonder if it actually works well. https://github.com/TecharoHQ/anubis https://github.com/TecharoHQ/anubis
- EPendragon 1y agoThis is a really cool tool. I haven't seen it before. Thank you for sharing it! On their README.md they state: > This program is designed to help protect the small internet from the endless storm of requests that flood in from AI companies. Anubis is as lightweight as possible to ensure that everyone can afford to protect the communities closest to them. I love the idea!
- mathgeek 1y ago> Trust by default, also by default, never ignoring suspicious signals. While I absolutely love the intent of this idea, it quickly falls apart when you're dealing with systems where you only get the signals after you've already lost everything of value.
- deleted 1y ago[deleted]
- 0xEF 1y agoIt's easy to believe, though, and most of us do it every day. For example, our commute to work is marked by the trust that other drivers will cooperate, following the rules, so that we all get to where we are going. There are varying degrees of this through our lives, where the trust lies not in the fact that people will just follow the rules because they are rules, but because the rules set expectations, allowing everyone to (more or less) know what's going on and decide accordingly. This also makes it easier to single out the people who do not think the rules apply to them so we can avoid trusting them (and, probably, avoid them in general).
- 6LLvveMx2koXfwn 1y agoI love the culturally specific implication that 'commute' == 'commute in the car' :)
- 0xEF 1y agoI apologize for that. I try to mitigate my US-centricness in my comments as much as possible, understanding completely that I am speaking with a global audience, but I am definitely not perfect at it :D I suppose the same goes if you take the tube, ride a bike, walk, etc? There's still rules in terms of behavior, flow of traffic (even foot traffic), etc, that helps set a number of expectations so everyone can decide and behave accordingly. Happy to hear different thoughts on this!
- pjmlp 1y agoIn Southern Europe, and countries with similar cultures, we don't obey rules because someone says so, we obey them when we see that is actually reasonable to do so, hence my remark regarding culture as I also experienced living in countries where everyone mostly blindly follow the rules, even if they happen to be nonsense. Naturally I am talking about cultures where that decision has not been taken away from their citizens.
- pmezard 1y agoAs a French, being passed by the right by Italian drivers on the highway really makes me feel the superiority of Southern Europeans judgment over my puny habit of blindly following rules. Or does it? But yes, I do the same. I just do not come here to pretend this is virtue.