8 ms·

I created a search engine that crawled the web way back in 2003. I used a proper user agent that included my email address. I got SO many angry emails about my

• tomrod 1y ago
  > But don’t just assume that every bot is malicious by default. I'll bite. It seems like a poor strategy to trust by default.
  • ronsor 1y ago
    I'll bite harder. That's how the public Internet works. If you don't trust clients at all, serve them a login page instead of content.
    • __loam 1y ago
      It sucks that we're living in a landscape where bad actors take advantage of that way of doing things.
      • sltkr 1y ago
        The really bad actors are going to ignore robots.txt entirely. You might as well be nice to the crawlers that respect robots.txt.
        • PeterStuer 1y ago
          Even if you want to play nice, robots.txt is a catch-22, as accessing it is taken as a signal you are a 'bot' by malconfigured anti-bot 'solutions'.
      • chasebank 1y ago
        Bad actors will always exploit whatever systems are available to them. Always have, always will.
      • KTibow 1y ago
        It sucks more that Cloudflare/similar have responded to this with "if your handshake fingerprints more like curl than like Chrome/Firefox, no access for you".
        • edoceo 1y ago
          Or getting a CAPTCHA from Chrome when visiting a site you've been to dozens of times (Stack Overflow). Now I just skip that content, probably in my LLM already anyway.
          • realusername 1y ago
            It's the same thing as the anti pirate ads, you only annoy legit customers, this agressive captcha campaign just makes Stackoverflow drop down even faster than it would normally by making it lower quality.
          • codingminds 1y ago
            Keep in mind that those LLMs are one of the bigger reasons why we see more and more anti bot behaviour on sites like SO. That aggressive crawling to train those on everything is insane.
        • NoMoreNicksLeft 1y ago
          I now write all of my bots in javascript and run them from the Chrome console with CORS turned off. It seems to defeat even Google's anti-bot stuff. Of course, I need to restart Chrome every few hours because of memory leaks, but it wasn't a fun 3 days the last time I got banned from their ecosystem with my kids asking why they couldn't watch Youtube.
          • tomrod 1y ago
            Where can I learn more about custom bots in JS and Chrome?
        • EPendragon 1y ago
          There are tools like curl-impersonate: https://github.com/lwthiker/curl-impersonate https://github.com/lwthiker/curl-impersonate out there that allow you to pretend to be any browser you like. Might take a bit of trial and error, but this mechanism could be bypassed with some persistence in identifying what is it that the resource is trying to block.
      • Perz1val 1y ago
        Because if they play by the rules, they won't be bad actors
    • tickettotranai 1y ago
      In fairness this appears to be the direction we are headed anyway
    • aopwiejfpoieajf 1y ago
      This is how it's going. Half the websites I go to have Cloudflare captchas guarding them at this point. Every time I visit StackOverflow I get a 5 second wait while Cloudflare decides I'm kosher.
      • Sohcahtoa82 1y ago
        Are you using TOR or a VPN, spoofing your User-Agent to something uncommon, or doing something else that tries to add extra privacy? That kind of user experience is one that I've seen a lot on HN, and every time, without fail, it's because they're doing something that makes them look like a bot, and then being all Surprised Pikachu when they get treated like a bot by websites.
        • EPendragon 1y ago
          I started having similar experiences when I switched to using Brave browser that blocks lots of tracking. Many websites that didn't show me those captchas and Cloudflare protection layers now have started to pop up on a regular basis.
        • ponooqjoqo 1y ago
          I get this and I assume it's because I clear cookies pretty frequently. It used to be the case that that didn't matter, but nowadays everyone shields their websites using JS.
        • 71bw 1y ago
          I use regular Edge with uBlock and get cloudflare crapchas all the time.
          • tomrod 1y ago
            I've found Vivaldi to be a much better experience, chrome based, and supports ublock in all its glory
            • 71bw 1y ago
• TylerE 1y ago
  That's easy to say when it's your bot, but I've been on the other side to know that the problem isn't your bot, it's the 9000 other ones just like it, none of which will deliver traffic anywhere close to the resources consumed by scraping.
  • kijin 1y ago
    True. Major search engines and bots from social networks have a clear value proposition: in exchange for consuming my resources, they help drive human traffic to my site. GPTBot et al. will probably do the same, as more people use AI to replace search. A random scraper, on the other hand, just racks up my AWS bill and contributes nothing in return. You'd have to be very, very convincing in your bot description (yes, I do check out the link in the user-agent string to see what the bot claims to be for) in order to justify using other people's resources on a large scale and not giving anything back. An open web that is accessible to all sounds great, but that ideal only holds between consenting adults. Not parasites.
    • NackerHughes 1y ago
      > GPTBot et al. will probably do the same, as more people use AI to replace search. It really won’t. It will steal your website’s content and regurgitate it back out in a mangled form to any lazy prompt that gets prodded into it. GPT bots are a perfect example of the parasites you speak of that have destroyed any possibility of an open web.
      • kijin 1y ago
        Only if the GPT companies can resist the temptation of all that advertising $$$. I'll give them at most 3 years before sponsored links begin appearing in the output and "AI optimization" becomes a fashionable service alongside the SEO snake oil. Most publishers won't care whether their content is mangled or not, as long as it is regurgitated with the right keywords and links.
        • tpxl 1y ago
• Jach 1y ago
  I guess back in 2003 people would expect an email to actually go somewhere, these days I would expect it to either go nowhere or just be part of a campaign to collect server admin emails for marketing/phishing purposes. Angry emails are always a bit much, but I wonder if they aren't sent as much anymore in general or if people just stopped posting them to point and laugh at and wonder what goes through people's minds to get so upset to send such emails. My somewhat silly take on seeing a bunch of information like emails in a user agent string is that I don't want to know about your stupid bot. Just crawl my site with a normal user agent and if there's a problem I'll block you based on that problem. It's usually not a permanent block, and it's also usually setup with something like fail2ban so it's not usually an instant request drop. If you want to identify yourself as a bot, fine, but take a hint from googlebot and keep the user agent short with just your identifier and an optional short URL. Lots of bots respect this convention. But I'm just now reminded of some "Palo Alto Networks" company that started dumping their garbage junk in my logs, they have the audacity to include messages in the user agent like "If you would like to be excluded from our scans, please send IP addresses/domains to: scaninfo@paloaltonetworks.com" or "find out more about our scans in [link]". I put a rule in fail2ban to see if they'd take a hint (how about your dumb bot detects that it's blocked and stops/slows on its own accord?) but I forgot about it until now, seems they're still active. We'll see if they stop after being served nothing but zipbombs for a while before I just drop every request with that UA. It's not that I mind the scans, I'd just prefer to not even know they exist.
  • EPendragon 1y ago
    I think a better solution would be to block all the traffic, but have a comment in robots.txt with a way to be added onto a whitelist to scrape the contents of the resource. This puts a burden of requesting the access on the owner of the bot, and if they really want that access, they can communicate it and we can work it out.
    • Jach 1y ago
      It's a nice option to have and maybe good in some cases. It reminds me of the nicety that some journalists do when requesting if they can use some video uploaded on social media for their show or piece. I do like the approach and shifting of first contact burden, as well as the general philosophical principle that blocking ought to be reversible and also temporary rather than permanent (though I also like the idea of exponential timeouts that can become effectively permanent). Still, I don't see myself ever doing anything like that. I'd still prefer to just not know about the bot at all, and if I did decide to perma-block them, unless the first contact comes with sufficient dollar signs attached I'm likely to ignore it entirely. I'm not usually in the mood for starting random negotiation with anybody. I also tend to see the web from the "open web" dream perspective. By default no traffic is blocked. The burden of requesting is already inherently done with a client -- they request a route, and I serve it or not. For things like my blog I don't tend to care who is requesting a particular route -- even admin pages can be requested, they just don't get anything without being logged in. If someone is being "cute" requesting non-existent wordpress pages or what have you, searching for vulnerabilities, or have an annoying/ugly user agent string, or are just pounding me for no real reason, then I do start to care. (The "pounding" aspect is a bit trickier -- I look at steady state. Another comment mentioned cutting their db server's cpu load in half by dropping unlikely-to-be-real-users from two countries. For me, if that is merely a steady state reduction from like 10% of a machine to 5%, I don't really care, I start caring when it would get in the way of real growth without having to use more resources.) When I was hosting on EC2, I used to have very mild anxiety that I'd piss off someone and they'd try to "harm" me by launching a botnet of requests at large media files and rack up bandwidth costs. (I believe it when some people say this has happened more organically with normal bots in the age of LLMs, but my concern was more targeted botnets/ddos.) There are a few ways to mitigate that anxiety: 1) setup monitoring, alerts, and triggers directly in code running on the instance itself or via overseeing AWS tools (I did the latter, which is less reliable, but still. There was a threshold to shutdown the whole instance, minimizing the total damage possible to something like under a couple hundred bucks, I forget the details of trying to calculate how much traffic could theoretically be served before the monitoring side noticed) 2) hide behind cloudflare and their unlimited bandwidth, as my content was mostly static (I didn't do that) 3) move/rearchitect to a free host like github pages, give up hosting my own comments (again didn't do) 4) move to OVH which has unlimited bandwidth (did this when Amazon wanted to start charging an absurd amount for just a single ipv4 address).
• mytailorisrich 1y ago
  It's just that people are suspicious of unknown crawlers, and rightly so. Since it is impossible to know a priori which crawler are malicious, and many are malicious, it is reasonable to default to considering anything unknown malicious.
• knorker 1y ago
  The most annoying thing about being a good bot owner, in my experience, is when you get complaints about it misbehaving, only to find that it was actually somebody malicious who wrote their own abusive bot, but is using your bot's user agent.
  • pimterry 1y ago
    Cloudflare have some new bot verification proposals designed to fix this, with cryptographic proofs that the user-agent is who they say they are: https://blog.cloudflare.com/web-bot-auth/ https://blog.cloudflare.com/web-bot-auth/.
    • EPendragon 1y ago
      It is awesome to see that there are efforts to improve upon the old standards with modern web security standards
• EPendragon 1y ago
  I definitely agree here. My initial response was to block everything, however you realize that web is complex and interdependent. I still believe that everyone should have autonomy over their online resources if they desire. But that comes with an intentionality behind it. If you want to allow or disallow certain traffic, you also should answer the question why or why not. That requires understanding what each bot does. That takes time and effort. My foray into robots.txt started from the whole notion of AI companies training on everything they can put their hands on. I want to be able to have a say whether I allow it or not. While not all bots will honor the robots.txt file, there are plenty that do. One way that I found you can test that is by asking the model directly to scrape a particular link (assuming the model has browsing capabilities). Bots are not malicious by default. It is what that company does with your data and how you feel about it that matters in the end.