4 ms·

I get that the author “doesn’t care anymore”, but I saw Google, ripe.net, and semrush in the banned IPs. Of these, I certainly wouldn’t ban Google, and probabl

I get that the author “doesn’t care anymore”, but I saw Google, ripe.net, and semrush in the banned IPs. Of these, I certainly wouldn’t ban Google, and probably not the others, if I wanted others to see it and talk about it. Even if your content were being scraped for some rando’s AI bot, why have a public site if you don’t expect your site to be used? Turning the lights off on the motel sign when you want people to find it is not a good way to invite people in.

DamonHD 1y ago
Semrush misbehaved so badly for such a long time with various levels of incompetence that I have special notes in my robots.txt files going back at least 8 years and eventually got someone in legal to pay attention. I see zero value to me or any potential users of mine in letting an 'SEO' firm lumpishly trample all over my site barging out real visitors. And some of Semrush's competitors were just as bad. The current round of AI nonsense also very poor. Again had to send legal notes to investor relations and PR depts in at least one well-known case, as well as all the technical measures, to restore some sort of decorum.
- akudha 1y ago
  My experience with them has been the same. On one of my employer's websites, the top three bots were Google, Bytedance and Semrush. It is a small website for a niche audience, not even in English, and changes very infrequently (like once or twice a quarter). That did not stop these three bots from hammering the site, every second
  - hinkley 1y ago
    We were getting hammered on nofollow links. Someone made a change to stop putting the nofollow in and using JavaScript to traverse on interaction, but that person left sometime after the toggle went in and we discovered it had never been turned on much later, when we were rearranging our ingress load balancer and needed to tweak how we handled bot throttling. That was at least 10% of the requests during peak traffic and extended the duration of the crawls by a lot. So dumb.
phyzome 1y ago
Because the bot requests are consuming significant amounts of bandwidth, memory, CPU, and disk space. Like the intro says, it's just rude, and there's no reason to serve traffic to harvesters like that. Google also runs an AI scraper, which might be what you saw represented there?
- globie 1y ago
  From the article it's sure starting to seem like people across the internet are just starting to realize what happens when you don't have just 3-4 search engines responsible for crawling for data anymore. When data becomes truly democratized, its access increases dramatically, and we can either adjust or shelter ourselves while the world moves on without us. Did Google never ever scrape individual commits from Gitea?
  - HumanOstrich 1y ago
    > When data becomes truly democratized... That is not at all what is happening.
    - globie 1y ago
      I know, we're locking everything down behind WAFs and repeating captchas so only attested identities can get access in the end.
      - HumanOstrich 1y ago
        Yep, everyone is building their own little walled gardens instead of adapting.
        itsafarqueue 1y ago
        Friend, this IS them adapting.
        1y ago
taormina 1y ago
There are also bad actors who pretend to be the Google scraper. Google once upon a time had a reputation for respectfully scrapping, but if he's getting the traffic he needs with or without the Googlebot, why should he care?
- hinkley 1y ago
  They had those stupid tags telling them far more anyway. Nevermind what it was doing to time to first paint/interactive.
BlueTemplar 1y ago
You haven't started educating people in the last ten years that it's just not acceptable to use Google any more ? Especially when this happens ? Google is using AI to censor independent websites like mine https://news.ycombinator.com/item?id=44124820 https://news.ycombinator.com/item?id=44124820 Sure sounds like we've reached the point where it's more of a liability !

DamonHD 1y ago

Semrush misbehaved so badly for such a long time with various levels of incompetence that I have special notes in my robots.txt files going back at least 8 years and eventually got someone in legal to pay attention. I see zero value to me or any potential users of mine in letting an 'SEO' firm lumpishly trample all over my site barging out real visitors. And some of Semrush's competitors were just as bad. The current round of AI nonsense also very poor. Again had to send legal notes to investor relations and PR depts in at least one well-known case, as well as all the technical measures, to restore some sort of decorum.

akudha 1y ago
My experience with them has been the same. On one of my employer's websites, the top three bots were Google, Bytedance and Semrush. It is a small website for a niche audience, not even in English, and changes very infrequently (like once or twice a quarter). That did not stop these three bots from hammering the site, every second
- hinkley 1y ago
  We were getting hammered on nofollow links. Someone made a change to stop putting the nofollow in and using JavaScript to traverse on interaction, but that person left sometime after the toggle went in and we discovered it had never been turned on much later, when we were rearranging our ingress load balancer and needed to tweak how we handled bot throttling. That was at least 10% of the requests during peak traffic and extended the duration of the crawls by a lot. So dumb.

phyzome 1y ago

Because the bot requests are consuming significant amounts of bandwidth, memory, CPU, and disk space. Like the intro says, it's just rude, and there's no reason to serve traffic to harvesters like that. Google also runs an AI scraper, which might be what you saw represented there?

globie 1y ago
From the article it's sure starting to seem like people across the internet are just starting to realize what happens when you don't have just 3-4 search engines responsible for crawling for data anymore. When data becomes truly democratized, its access increases dramatically, and we can either adjust or shelter ourselves while the world moves on without us. Did Google never ever scrape individual commits from Gitea?
- HumanOstrich 1y ago
  > When data becomes truly democratized... That is not at all what is happening.
  - globie 1y ago
    I know, we're locking everything down behind WAFs and repeating captchas so only attested identities can get access in the end.
    - HumanOstrich 1y ago
      Yep, everyone is building their own little walled gardens instead of adapting.
      - itsafarqueue 1y ago
        Friend, this IS them adapting.
        1y ago

taormina 1y ago

There are also bad actors who pretend to be the Google scraper. Google once upon a time had a reputation for respectfully scrapping, but if he's getting the traffic he needs with or without the Googlebot, why should he care?

hinkley 1y ago
They had those stupid tags telling them far more anyway. Nevermind what it was doing to time to first paint/interactive.

BlueTemplar 1y ago

You haven't started educating people in the last ten years that it's just not acceptable to use Google any more ? Especially when this happens ? Google is using AI to censor independent websites like mine https://news.ycombinator.com/item?id=44124820 https://news.ycombinator.com/item?id=44124820 Sure sounds like we've reached the point where it's more of a liability !