5 ms·

This guy has had so many issues with various crawlers "attacking" his (boring!) sites, I'm starting to think he might just be a bit paranoid. see: https://www

This guy has had so many issues with various crawlers "attacking" his (boring!) sites, I'm starting to think he might just be a bit paranoid. see: https://www.hackerfactor.com/blog/index.php?/archives/678-Bot-Spotting.html https://www.hackerfactor.com/blog/index.php?/archives/678-Bo... https://www.hackerfactor.com/blog/index.php?/archives/762-Attacked-Over-Tor.html https://www.hackerfactor.com/blog/index.php?/archives/762-At... https://www.hackerfactor.com/blog/index.php?/archives/775-Scans-and-Attacks.html https://www.hackerfactor.com/blog/index.php?/archives/775-Sc... https://www.hackerfactor.com/blog/index.php?/archives/777-Stopping-Tor-Attacks.html https://www.hackerfactor.com/blog/index.php?/archives/777-St... https://www.hackerfactor.com/blog/index.php?/archives/779-Behind-the-Tor-Attacks.html https://www.hackerfactor.com/blog/index.php?/archives/779-Be... I've done lots of portscanning of the entire internet, this always generates vast amounts of abuse emails. Many of these these emails are xenophobic rants from obviously distressed people staring at their access.logs, thinking they're being targeted by the Russian government or something. I can't help but wonder if this might be a similar case.

hnabz 8y ago
The imgur thing looks like Google was following links from another webpages. i.e. I submit a pic to fotoforensics, I post the link here (something like fotoforensics.com/analyze?url=imgur.com/blabla) and Google ended up crawling all those links in batches. The fact that he says something like "a Google administrator tracked down a rogue employee and made him stop" makes it clear he's paranoid (or malicious).
- wybiral 8y ago
  > The fact that he says something like "a Google administrator tracked down a rogue employee and made him stop" makes it clear he's paranoid (or malicious). Or sensationalism. (it did land a top spot on HN)
- mintplant 8y ago
  If you read the original post about that issue [0], the imgur.com URLs submitted were for imgur web pages containing an image, rather than direct image URLs. This caused the FotoForensics service to emit an error message. Users wouldn't have any reason to link to those error pages. [0] https://www.hackerfactor.com/blog/index.php?/archives/484-Google-Abuse.html https://www.hackerfactor.com/blog/index.php?/archives/484-Go...
golergka 8y ago
Curiously, I've seen similar posts in various hacker blogs a couple of times. It's typically a very stereotypical hacker, running his own website from a machine that he personally administers, that runs only free software. Because his (and, unsurprisingly, all of them are men) other hobbies are very specific and rare, and he's very competent in them, he getting a reasonable amount of search traffic, but never a lot of comments. I've never seen anyone who doesn't fit this stereotype posting analysis of URL hits. It's kind of seems that only this kind of person (which I actually like a lot — these guys are very fun when you have common interests) would be so anal about this.
- hnabz 8y ago
  Nice sexist message my friend.
- stevew20 8y ago
  Yeah, sexist comments are what female empowerment is all about!
bitL 8y ago
On one of my obscure but public domains I always have fun seeing Chinese traffic guessing the admin password of the VPS at least 3x a day. I guess somebody is continuously poking at vulnerabilities to get root access to as many domains as they can, then later maybe selling them to their "customers"? One doesn't have to be paranoid these days...
- ryanlol 8y ago
  Anybody with a gbit line will be able to easily bruteforce the entire internet, there are lots of people doing that. This is just normal background noise on the internet, one has to be pretty paranoid to assign any significance to it.
- wybiral 8y ago
  True, you'll get all kinds of weird traffic like that running public services. Especially hidden services on Tor where people generally aren't backed by CDNs and attacks can be made anonymously. But it's weirder to claim that Googlebot is aggressively DDoSing you.
falcolas 8y ago
Well, there's always the saying "It's not paranoia if they're really out to get you." And DOS attacks of many flavors are employed early and often against any public site. And regardless of the cause, someone at Google has messed up by either letting themselves be used or screwing up the crawling algorithms to create so much load against one fairly minor blog.
- jonas21 8y ago
  He's complaining that Google is sending 100 requests per hour to his site (i.e. one request every 36 seconds). That's hardly a DOS attack. I doubt if anyone would even notice, except that he has his server set up to alert on every 404. Sure sounds like paranoia to me.
  - falcolas 8y ago
    No, he's complaining that they're generating 100 alerts per hour. And that Google crawlers consume over 25% of his total available resources. That's potentially a lot of money wasted on nothing.
    - jonas21 8y ago
      The reason it's generating 100 alerts per hour is that his server alerts on every request to a "bad" (non-existent?) URL: > My blog logs an alert when someone accesses a bad URL. The reason for the alert: a series of bad URL requests is often an indicator of an attack, or a precursor to an attack. And he doesn't say it's using 25% of his available resources -- he says 25% of visits to his blog are from Googlebot. Since he doesn't mention what his total visits are, we have no way of knowing how much money is wasted, but I'd be willing to bet the amount is pretty close to $0.00.
deleted 8y ago
[deleted]

hnabz 8y ago

The imgur thing looks like Google was following links from another webpages. i.e. I submit a pic to fotoforensics, I post the link here (something like fotoforensics.com/analyze?url=imgur.com/blabla) and Google ended up crawling all those links in batches. The fact that he says something like "a Google administrator tracked down a rogue employee and made him stop" makes it clear he's paranoid (or malicious).

wybiral 8y ago
> The fact that he says something like "a Google administrator tracked down a rogue employee and made him stop" makes it clear he's paranoid (or malicious). Or sensationalism. (it did land a top spot on HN)
mintplant 8y ago
If you read the original post about that issue [0], the imgur.com URLs submitted were for imgur web pages containing an image, rather than direct image URLs. This caused the FotoForensics service to emit an error message. Users wouldn't have any reason to link to those error pages. [0] https://www.hackerfactor.com/blog/index.php?/archives/484-Google-Abuse.html https://www.hackerfactor.com/blog/index.php?/archives/484-Go...

golergka 8y ago

Curiously, I've seen similar posts in various hacker blogs a couple of times. It's typically a very stereotypical hacker, running his own website from a machine that he personally administers, that runs only free software. Because his (and, unsurprisingly, all of them are men) other hobbies are very specific and rare, and he's very competent in them, he getting a reasonable amount of search traffic, but never a lot of comments. I've never seen anyone who doesn't fit this stereotype posting analysis of URL hits. It's kind of seems that only this kind of person (which I actually like a lot — these guys are very fun when you have common interests) would be so anal about this.

hnabz 8y ago
Nice sexist message my friend.
stevew20 8y ago
Yeah, sexist comments are what female empowerment is all about!

bitL 8y ago

On one of my obscure but public domains I always have fun seeing Chinese traffic guessing the admin password of the VPS at least 3x a day. I guess somebody is continuously poking at vulnerabilities to get root access to as many domains as they can, then later maybe selling them to their "customers"? One doesn't have to be paranoid these days...

ryanlol 8y ago
Anybody with a gbit line will be able to easily bruteforce the entire internet, there are lots of people doing that. This is just normal background noise on the internet, one has to be pretty paranoid to assign any significance to it.
wybiral 8y ago
True, you'll get all kinds of weird traffic like that running public services. Especially hidden services on Tor where people generally aren't backed by CDNs and attacks can be made anonymously. But it's weirder to claim that Googlebot is aggressively DDoSing you.

falcolas 8y ago

Well, there's always the saying "It's not paranoia if they're really out to get you." And DOS attacks of many flavors are employed early and often against any public site. And regardless of the cause, someone at Google has messed up by either letting themselves be used or screwing up the crawling algorithms to create so much load against one fairly minor blog.

jonas21 8y ago
He's complaining that Google is sending 100 requests per hour to his site (i.e. one request every 36 seconds). That's hardly a DOS attack. I doubt if anyone would even notice, except that he has his server set up to alert on every 404. Sure sounds like paranoia to me.
- falcolas 8y ago
  No, he's complaining that they're generating 100 alerts per hour. And that Google crawlers consume over 25% of his total available resources. That's potentially a lot of money wasted on nothing.
  - jonas21 8y ago
    The reason it's generating 100 alerts per hour is that his server alerts on every request to a "bad" (non-existent?) URL: > My blog logs an alert when someone accesses a bad URL. The reason for the alert: a series of bad URL requests is often an indicator of an attack, or a precursor to an attack. And he doesn't say it's using 25% of his available resources -- he says 25% of visits to his blog are from Googlebot. Since he doesn't mention what his total visits are, we have no way of knowing how much money is wasted, but I'd be willing to bet the amount is pretty close to $0.00.

deleted 8y ago

[deleted]