4 ms·

I've still never understood the complaint here. Robots are part of the web, the whole point of HTML is that robots can read it.

• marginalia_nu 1y ago
  It's a cost thing. It costs more to render a website than it does to consume it. When you have some bot traffic mixed in with human traffic, that is fine. When you have egregious bot traffic, say 10k requests per minute sustained load, it becomes a real problem for webmasters.
• pjc50 1y ago
  The whole point of publication is so that humans can read it. Robots not so much, especially if they're not paying customers. This is the distinction between how the web works technically and how it works socioeconomically. This is the next iteration of things like the news snippet case. Publishers are not happy that Google crawls their content (at their expense) and then republishes it on their own site, while serving ads around it and getting user data, without cutting in the publisher who originally made it. And, for what little it's worth, owns the copyright.
  • msgodel 1y ago
    Robots don't exist for their own sake, at the end of the day they are user agents for some group of humans. Again it sounds like the people who are upset by this really want to publish images rather than web pages.
    • imilk 1y ago
      > Again it sounds like the people who are upset by this really want to publish images rather than web pages. More like people don't want to lose money because a 3rd party stole all of their content, and then repurposed it to show people before they visit their website.
  • CaptainFever 1y ago
    Web 3.0 is all about machine-readability: https://en.m.wikipedia.org/wiki/Semantic_Web https://en.m.wikipedia.org/wiki/Semantic_Web (Not to be confused with Web3.)
• pluto_modadic 1y ago
  HTML was not made for robots to read (a semantic web or an internet of data), it just so happens that crawlers try to index things in meaningful ways. It's an un-ordered blob of unstructured data.
• josefresco 1y ago
  Have you ever been responsible for the performance and security of a publicly accessible web server? I'll accept robots indexing my content if they play nicely. Unfortunately most do not, even from major vendors.
  • msgodel 1y ago
    Not a web server but yeah we dealt with it by black listing patterns (IPs, requests etc) from misbehaving domains. We never distinguished automations from people though, that makes no sense on the internet.
    • imilk 1y ago
      > We never distinguished automations from people though, that makes no sense on the internet. LOL I see you've never sold anything on the internet, ran a website that is supposed to generate leads, or had to gauge the effectiveness of an ad campaign. There is a huge part of the internet that relies on real humans doing things on websites. And ignoring that is insane.
      • msgodel 1y ago
        Maybe I should clarify: It makes no sense from a system architecture perspective. Obviously if you're doing analytics you want to know the difference.
        • imilk 1y ago
          A systems architecture perspective should be very very inclusive of the business perspective.
          • msgodel 1y ago
            In theory. In practice business people have such a poor intuition for systems design you end up with borderline unusable software when you do that.
• mattl 1y ago
  A lot of people don't want AI slop and don't want the companies pushing it to crawl their websites.
• dehrmann 1y ago
  That's why we need the Captchaweb. It's like the web, but everything is in captcha text.
• imilk 1y ago
  Having perplexity or other AI bots go haywire and sending 10s of thousands of requests per minute to your website (despite you having a robots.txt blocking them) is a giant pain in the ass. Not only does your server costs go up, but your analytics and attribution reports start to look messed up because of all the bot traffic.
  • msgodel 1y ago
    Yeah obviously if they're being abusive that's a problem but that's not what the article seems to be talking about.
    • imilk 1y ago
      Well besides them being abusive, the other issue is that AI overviews and answer boxes cannibalize traffic to websites, leading to less conversions for the original content producers. This is pretty well established across industries at this point: https://ahrefs.com/blog/ai-overviews-reduce-clicks/ https://ahrefs.com/blog/ai-overviews-reduce-clicks/
      • msgodel 1y ago
        That's how people want to browse the web. If you block it you won't even get links from those. That's like blocking the search crawler.
        • imilk 1y ago
          ...that's literally the entire point of this article. People don't want their websites being de-listed from the monopoly that controls organic traffic. At the same time they would like some control over stopping companies (in this case, the same company that controls the organic search monopoly) from scraping and repurposing their content so their the traffic to their website doesn't decrease. Why is it such an issue that publishers and website owners want to maintain the traffic to their website so that they can continue operating as usual? Or should we all just accept every Google decision, even when those decisions result in more engagement on google.com, but 20-35% decreases in traffic to the original websites? Also I'm going to need a citation that the vast majority of people want and get value out of AI overviews. Because that is certainly not the case from my experience.
• eviks 1y ago
  Because you remain at the top of an extremely shallow perspective. The fact that robots are a part is not relevant, the relevant issue is how some of those robots behave, and what the consequences of that behavior are.
• alganet 1y ago
  Machine-readable does not mean centralized.